Durian maturity detection method based on improved YOLOv8n and multi-mode large model

Abstract

The invention discloses a durian maturity detection method based on improved YOLOv8n and a multi-mode large model. The durian maturity detection method comprises the following steps: acquiring durian image data sets of different maturity; randomly dividing the data set into a training set, a verification set and a test set, and performing data enhancement on the training set; marking immature and mature states of the durians; a coordinate attention mechanism is embedded in the YOLOv8 model, and a CA-YOLOv8 model is constructed; respectively using the training set and the verification set to train and optimize a CA-YOLOv8 model; and inputting a detection result of the CA-YOLOv8 model into the multi-mode large model, and generating a comprehensive analysis report of durian maturity detection. According to the method, intelligent judgment and multi-dimensional information output of durian maturity can be realized, the hallucination phenomenon in large model output is effectively inhibited while the detection precision is improved, and the credibility of a result is enhanced.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and agricultural information technology, specifically relating to a method for detecting durian maturity based on an improved YOLOv8n and a multimodal large model. Background Technology

[0002] As a tropical fruit with high economic value, accurately determining the ripeness of durian is crucial for harvesting, storage, transportation, and sales. Traditional ripeness testing relies mainly on human experience, judging ripeness by observing the color of the peel, the shape of the thorns, and the sound when tapped. This method suffers from high subjectivity, low efficiency, high cost, and susceptibility to individual factors, making it difficult to meet the needs of modern large-scale and intelligent agriculture.

[0003] In recent years, computer vision-based detection technologies, especially deep learning object detection models (such as YOLO and SSD), have been attempted to be applied to the automated detection of fruit external quality. These methods extract features from images to locate and classify targets, improving detection efficiency to some extent. However, when applied to fruits like durian, which have complex structures and subtle ripening characteristics, existing models suffer from limited receptive fields, long-range dependence, and insufficient ability to capture subtle features, resulting in low detection accuracy and difficulty in meeting practical requirements.

[0004] Furthermore, although multimodal large models have demonstrated powerful capabilities in general visual understanding tasks, their direct application to professional agricultural inspection scenarios has significant limitations: their generated results often exhibit illusion phenomena, that is, they produce descriptions that seem reasonable but do not conform to the facts of the image, which seriously affects the credibility of applications in scenarios with high reliability requirements.

[0005] Therefore, there is a lack of intelligent detection solutions for durian ripeness that can balance high-precision perception and reliable reasoning analysis in the existing technology. Summary of the Invention

[0006] The main objective of this invention is to overcome the shortcomings and deficiencies of the existing technology and provide a durian maturity detection method based on an improved YOLOv8n and a multimodal large model. This method enables intelligent judgment of durian maturity and multi-dimensional information output, while improving detection accuracy and effectively suppressing the illusion phenomenon in the output of the large model, thereby enhancing the credibility of the results.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: A method for detecting durian maturity based on an improved YOLOv8n and a multimodal large model, the method comprising the following steps: S1. Obtain a dataset of durian images at different maturity levels from the Roboflow Universe platform, wherein the Roboflow Universe platform is a public image dataset platform that provides rich image resources and convenient data management functions; S2. Randomly divide the dataset into training, validation, and test sets, and perform data augmentation on the training set; S3. Use the LabelImg tool to label the unripe and ripe states of durians. Hereinafter, the unripe state is abbreviated as Unripe and the ripe state is abbreviated as Ripe. The LabelImg tool is an open-source tool for image labeling, which is widely used in target detection tasks in the field of computer vision. S4. Embed coordinate attention mechanism into YOLOv8 model to construct CA-YOLOv8 model. The YOLOv8 model is the next major update of YOLOv5, which was open-sourced by Ultralytics on January 10, 2023. It performs well in image classification, object detection and instance segmentation tasks. S5. Train and optimize the CA-YOLOv8 model using the training set and validation set respectively; S6. Input the detection results of the CA-YOLOv8 model into the multimodal large model HuaNong_Vision to generate a comprehensive analysis report on durian maturity detection. The multimodal large model HuaNong_Vision is a professional multimodal large model for the agricultural field. Its core capability lies in the deep integration of visual information and text knowledge to achieve intelligent cognition and decision support in agricultural scenarios. That is, it supports inputting image information and text information in the agricultural field and outputting a comprehensive analysis report on agricultural scenarios.

[0008] Furthermore, step S1 is performed as follows: S1.1 Using the Roboflow Universe platform, search using keywords related to durian maturity to obtain a publicly available dataset of durian images at different maturity stages. This method can efficiently collect durian images from different scenes, lighting conditions, and shooting angles. S1.2. Filter the acquired images, removing blurry, duplicate, or unclear images to ensure the quality and validity of the dataset.

[0009] Furthermore, step S2 is as follows: S2.1 After randomly shuffling the dataset, divide it into training set, validation set and test set in a ratio of 7:2:1; S2.2. Perform data augmentation on the training set, including flipping, scaling, centering, adding noise, and tone transformation, to expand the training set samples and improve the model's generalization ability.

[0010] Furthermore, step S3 is as follows: S3.1 Use the LabelImg tool to label each individual durian in the image with a rectangular box. Label the green and hard durian as Unripe, and label the yellow-green, golden yellow or yellowish-brown durian with natural cracks in the shell as Ripe, to ensure the consistency and repeatability of the labeling results. S3.2 Save the annotation results of each image in the dataset in YOLO format, and generate a corresponding txt text file with the same name for each image file.

[0011] Furthermore, step S4 is as follows: S4.1 To address the shortcomings of standard convolutional operations in capturing long-range spatial dependencies and effectively perceiving the global contextual information of a durian in different directions within an image, this step uses the initial feature map output from the backbone network of the YOLOv8 model as input. Global average pooling is then performed along both the width and height spatial dimensions, decomposing the two-dimensional global spatial information into a pair of direction-aware feature vectors. This provides the model with the ability to capture long-range dependencies along a single spatial direction. , , in, and These are feature vectors perceived in the width and height directions, respectively. and The width and height dimensions are respectively, F() represents the input feature map, i and j are the loop variables for traversing and summing in the height and width directions, and w and h are the position indices of the vector in the width and height directions, respectively. S4.2 To effectively fuse global information from two directions and introduce nonlinearity to enhance representation capabilities while controlling computational complexity, this step will... and The data is then concatenated, followed by channel dimensionality reduction using a 1×1 convolution kernel, reducing the number of channels to 1 / r of the original. After batch normalization and ReLU activation, intermediate feature maps are generated. This ensures that the attention weights generated subsequently accurately reflect the differences in feature importance at different spatial locations, providing crucial spatial encoding information for precisely locating the ripening feature regions of durian fruits. , in, This represents the generated intermediate feature map, where C is the number of channels in the feature map. This represents the channel reduction ratio. () is the ReLU activation function, and BN() is the batch normalization operation. It is a 1×1 convolution kernel. For splicing operations; S4.3 To enable the model to learn the importance distribution in spatial location and adaptively weight features at different coordinate locations, this step splits the intermediate feature map f into two feature maps along the spatial dimension, and restores their channel count to 1×1 using a 1×1 convolution kernel. Then, by using the Sigmoid activation function, width attention weights and height attention weights are generated, which can accurately represent the importance of each position in the image in the vertical and horizontal directions: , , in, and These are the width attention weights and the height attention weights, respectively. () represents the Sigmoid activation function. and These represent the feature maps after splitting the intermediate feature map f along the width and height dimensions, respectively. and They respectively represent the effects on and A 1×1 convolution kernel; S4.4 To effectively apply directional attention weights to the original features, thereby enhancing key region features and suppressing unimportant background or interfering features, this step applies high attention weights. and width attention weight Element-wise multiplication is performed with the initial feature map F to achieve coordinate-aware weighting of the original features, and the enhanced feature map is output. This feature map can more prominently display the key features of durian (such as the main body and areas of color and texture variation), significantly improving the model's ability to distinguish subtle differences in ripeness. Here, F represents the initial feature map, which is the abbreviation for the input feature map F().

[0012] Furthermore, step S5 is as follows: The CA-YOLOv8 model was iteratively optimized using the training set to enable it to learn the essential characteristics of durian ripeness. Forward inference was performed using the validation set. By continuously monitoring the changing trends of key indicators such as loss function, precision, recall, and mean precision, the model parameters at the moment when the validation set performance reached its optimal level and showed a stable downward trend were saved as the best CA-YOLOv8 model. This ensured that the selected model had the strongest generalization ability and provided a reliable experimental basis for subsequent fair performance comparisons with the benchmark model. Thus, the actual performance improvement effect of the coordinate attention mechanism on the model was objectively verified.

[0013] Furthermore, step S6 is as follows: S6.1 The CA-YOLOv8 model is used to detect the images in the test set. The information on the location, category confidence and maturity status of durians is used as input to the multimodal large model HuaNong_Vision. This provides accurate and verifiable visual evidence for the multimodal large model HuaNong_Vision, cutting off the possibility of incorrect inference caused by erroneous visual information from the source. S6.2 The multimodal large model HuaNong_Vision, combined with its internal prior knowledge, performs contextual understanding and reasoning on the detection results, generating a comprehensive analysis report that includes maturity score (0-10 points), durian variety, consumption suggestions, storage guidelines, and explanations of key judgment criteria. This effectively suppresses the "illusion" problem that large models often encounter in professional fields, ensuring the authenticity and credibility of the final analysis report.

[0014] Furthermore, the durian maturity detection method also includes a step of comparing and evaluating the performance indicators and detection results of the YOLOv8 model and the CA-YOLOv8 model, as follows: S5.1. The YOLOv8 model and CA-YOLOv8 model were trained using the training set respectively, and the training process was monitored using the validation set to prevent overfitting. At the same time, the best model was selected based on the performance indicators on the validation set, ensuring that the "best model" with the strongest generalization ability under their respective architectures was used for subsequent comparisons, thus laying the foundation for fair and effective performance comparison. S5.2: After training, the performance metrics and detection results of the selected best YOLOv8 model and the best CA-YOLOv8 model are compared and evaluated using the test set. This intuitively demonstrates the advantages of the CA-YOLOv8 model over the YOLOv8 model in terms of detection accuracy and robustness, thereby verifying the effectiveness and technical value of the core improvement points of this invention. S5.3: Calculate the performance indicators using the following formulas respectively: , , , , Where Precision represents accuracy, TP represents the number of correctly predicted samples, and TP+FP represents the total number of predicted samples; Recall represents recall, and TP+FN represents the actual number of positive samples. denoted by p, which represents the average precision of the category predictions, and r, which represents the recall rate; mAP represents the mean average precision of all category predictions, and N represents the total number of categories.

[0015] Compared with the prior art, the present invention has the following advantages and beneficial effects: 1. This invention embeds a coordinate attention mechanism into the YOLOv8 model, enabling it to encode global dependencies along both the height and width directions. This design allows the model to accurately capture key distinguishing features such as peel color gradation and the distribution of fruit spine morphology during the recognition process, thus maintaining a high recognition accuracy even under complex backgrounds, varying lighting conditions, and partial fruit occlusion. Performance comparison results in Example 1 show that the improved CA-YOLOv8 achieves an average precision improvement of approximately 1.2 percentage points in the "Comprehensive" category on the test set, and a recall improvement of over 6% in the "Mature" category.

[0016] 2. This invention creatively combines a high-precision target detection model with a multimodal large model, utilizing the accurate detection results (such as location, category, and confidence level) provided by CA-YOLOv8 as reliable factual evidence to guide and constrain the generation process of the multimodal large model HuaNong_Vision. This approach suppresses the "illusion" problem commonly seen in large models in specialized fields, ensuring the authenticity and credibility of the comprehensive analysis report. 3. Traditional detection methods can only provide a binary classification of "mature / immature," which cannot meet the needs of decision support information in actual production. This invention leverages the powerful knowledge fusion and natural language generation capabilities of the multimodal large model HuaNong_Vision to transform the detection results of CA-YOLOv8 into a comprehensive analysis report that includes quantitative scoring, variety inference, storage guidance, and key judgment criteria. This analysis report not only provides "what" but also explains "why," for example, by combining the detection box position and color confidence to explain the reasons for maturity judgment, greatly enhancing the interpretability of the results and providing growers, buyers, and other users with directly usable decision-making basis. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of the overall process of the durian maturity detection method based on the improved YOLOv8n and multimodal large model in an embodiment of the present invention; Figure 2 This is a diagram illustrating the data enhancement effect of an embodiment of the present invention; Figure 3 This is a schematic diagram of the overall structure of the CA-YOLOv8 detection model according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the module structure of the coordinate attention mechanism in the CA-YOLOv8 detection model of this invention. Figure 5 This is a schematic diagram showing the detection comparison results of the YOLOv8 model and the CA-YOLOv8 model on the test set according to an embodiment of the present invention; Figure 6 This is a schematic diagram of the comprehensive analysis report results after the detection results of the CA-YOLOv8 model in this embodiment of the invention are input into the multimodal large model HuaNong_Vision. Detailed Implementation

[0019] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0020] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0021] Example 1 like Figure 1 As shown in the figure, this embodiment discloses a method for detecting durian maturity based on an improved YOLOv8n and a multimodal large model, which mainly includes the following steps: S1. Obtain a dataset of durian images at different maturity levels from the Roboflow Universe platform, wherein the Roboflow Universe platform is a public image dataset platform that provides rich image resources and convenient data management functions; S2. Randomly divide the dataset into training, validation, and test sets, and perform data augmentation on the training set; S3. Use the LabelImg tool to label the unripe and ripe states of durians. Hereinafter, the unripe state is abbreviated as Unripe and the ripe state is abbreviated as Ripe. The LabelImg tool is an open-source tool for image labeling, which is widely used in target detection tasks in the field of computer vision. S4. Embed coordinate attention mechanism into YOLOv8 model to construct CA-YOLOv8 model. The YOLOv8 model is the next major update of YOLOv5, which was open-sourced by Ultralytics on January 10, 2023. It performs well in image classification, object detection and instance segmentation tasks. S5. Train and optimize the CA-YOLOv8 model using the training set and validation set respectively; S6. Input the detection results of the CA-YOLOv8 model into the multimodal large model HuaNong_Vision to generate a comprehensive analysis report on durian maturity detection. The multimodal large model HuaNong_Vision is a professional multimodal large model for the agricultural field. Its core capability lies in the deep integration of visual information and text knowledge to achieve intelligent cognition and decision support in agricultural scenarios. That is, it supports inputting image information and text information in the agricultural field and outputting a comprehensive analysis report on agricultural scenarios.

[0022] The following is a detailed explanation of each of the above steps: In step S1 above, this invention selects the Roboflow Universe platform as the core data source. Using the search interface provided by the Roboflow Universe platform, keywords related to durian ripeness such as "durian," "ripedurian," and "unripe durian" are used to systematically obtain relevant publicly available image datasets from within the platform. This method can efficiently aggregate durian images from different scenes, lighting conditions, and shooting angles.

[0023] In this embodiment, a strict quality control process is implemented on the initially acquired image data. First, images with serious quality problems are removed through manual visual inspection or automated scripts, such as blurry images, images where the main body of the durian is severely obscured, and images containing a large number of irrelevant distractions. Second, to avoid data redundancy and ensure data validity, images with highly repetitive content and those whose ripeness status cannot be clearly determined need to be removed.

[0024] In step S2 above, a fixed random seed is set before splitting the dataset, and the complete dataset is randomly shuffled. It is then divided into training, validation, and test sets in a 7:2:1 ratio.

[0025] In this embodiment, data augmentation is performed on the training set, including flipping, scaling, center cropping, adding noise, and tone transformation. This data augmentation significantly improves the visual diversity of the training samples, effectively enhancing the robustness and generalization performance of the model. The data augmentation effect is shown in the figure below. Figure 2 As shown, where: Figure 2 (a) represents the original image selected from the dataset; (b) represents the data augmentation method of horizontal flipping; (c) represents the data augmentation method of shrinking; (d) represents the data augmentation method of center cropping; (e) represents the data augmentation method of adding noise; and (f) represents the data augmentation method of tone transformation.

[0026] In step S3 above, based on the visual characteristics of durian ripeness, the LabelImg tool is used to perform fine labeling on each image in the dataset: durians with greenish-blue peel, large angle of the thorns and hard texture are labeled as Unripe; durians with yellowish-green, golden yellow or yellowish-brown peel, slightly loose thorns and natural cracks in the shell are labeled as Ripe.

[0027] In this embodiment, the annotation results are uniformly converted to YOLO format for saving, and a corresponding .txt text file with the same name is generated for each image file. The file content contains the normalized coordinate information of each annotated target, in the following format:<class_id><x_center><y_center> <width> <height>After the annotation is completed, sample verification is performed to ensure that the annotation content is consistent with the actual content of the image, and the coordinate format meets the YOLO training requirements.

[0028] In the above step S4, the YOLOv8 model is improved, and a coordinate attention mechanism (Coordinate Attention) is embedded in the YOLOv8 model to construct a CA-YOLOv8 model. The overall structure diagram of the CA-YOLOv8 model is shown in Figure 3 The module structure diagram of the coordinate attention mechanism is shown in Figure 4 The specific steps are as follows: S41, the initial feature map with dimensions CxHxW output by the YOLOv8 backbone network is input as Input, and X Avg Pool and Y Avg Pool global average pooling are performed along the width and height two spatial dimensions respectively to generate a pair of feature vectors with dimensions CxHx1 and Cx1xW respectively: , , wherein, and are the width direction and height direction perception feature vectors, and are the width dimension and height dimension, F() represents the input feature map, i and j are the loop variables for traversing and summing in the height direction and width direction respectively, and w and h are the position indexes of the vector in the width direction and height direction respectively; S42, the feature vectors obtained in step S41 are and are concatenated, and then a 1x1 Conv2d convolution kernel is used for channel dimension reduction, the channel number is reduced to 1 / r of the original, and then batch normalization BatchNorm processing is performed and passed through a ReLU activation function to generate an intermediate feature map with dimensions C / rx1x(H+W): , wherein, represents the generated intermediate feature map, C is the channel number of the feature map, is the channel reduction ratio, () is the ReLU activation function, BN() is the batch normalization operation, is a 1x1 convolution kernel, is a concatenation operation; S43, the intermediate feature map f obtained in step S42 is split into two feature maps along the spatial dimension, and the channel number of each is restored to Then, through the Sigmoid activation function, attention weights with dimensions C×H×1 and C×1×W are generated: , , in, and These are the width attention weights and the height attention weights, respectively. () represents the Sigmoid activation function. and These represent the feature maps after splitting the intermediate feature map f along the width and height dimensions, respectively. and They respectively represent the effects on and A 1×1 convolution kernel; S44. Apply the high attention weights obtained in step S43. and width attention weight Element-wise multiplication is performed with the initial feature map F to achieve coordinate-aware weighting of the original features, outputting an enhanced feature map with dimensions C×H×W. , where F represents the initial feature map, which is short for the input feature map F().

[0029] In step S5 above, the deep learning framework PyTorch 1.12.1, the GPU acceleration library CUDA 11.3, the development platform PyCharm, the programming language Python 3.8.20, and the operating system Ubuntu 18.04 are used; the hardware configuration uses an Intel(R) Xeon(R) Platinum 8255C CPU @2.50GHz processor and an NVIDIA GeForce RTX 4090 graphics card with 24564MiB.

[0030] In this embodiment, the training parameters of the model are set, and the training parameters and parameter values ​​are shown in Table 1.

[0031] Table 1. Model Training Parameters and Parameter Values

[0032] In this embodiment, the performance indicators are calculated using the following formulas: , , , , Where Precision represents accuracy, TP represents the number of correctly predicted samples, and TP+FP represents the total number of predicted samples; Recall represents recall, and TP+FN represents the actual number of positive samples. denoted by p, which represents the average precision of the category predictions, and r, which represents the recall rate; mAP represents the mean average precision of all category predictions, and N represents the total number of categories.

[0033] In this embodiment, the CA-YOLOv8 model is trained with the training set for 200 iterations, and the training process is monitored using the validation set to prevent overfitting. At the same time, the best model CA-YOLOv8 is selected based on the performance metrics on the validation set, laying the foundation for a fair comparison with the YOLOv8 model in the future.

[0034] In step S6 above, the CA-YOLOv8 model detects the test set images, and the obtained information on durian location, category confidence, and maturity status is used as input to the multimodal large model HuaNong_Vision. This provides accurate and verifiable visual evidence for the multimodal large model HuaNong_Vision, cutting off the possibility of erroneous inference caused by erroneous visual information at the source.

[0035] In this embodiment, the HuaNong_Vision model, combined with its internal prior knowledge, performs contextual understanding and reasoning on the detection results to generate a comprehensive analysis report that includes a maturity score (0-10 points), durian variety, consumption recommendations, storage guidelines, and explanations of key judgment criteria. Figure 6 As shown, this comprehensive analysis report on durian ripeness detection represents an intelligent upgrade from simple visual detection to one with semantic understanding and decision support.

[0036] In this embodiment, the YOLOv8 model and the CA-YOLOv8 model were trained for 200 iterations using the training set, respectively. Based on the overall performance of the models on the validation set (mainly referring to the average precision), the best weight file (best.pt) of YOLOv8 and CA-YOLOv8 on the validation set was selected as the final model. To objectively evaluate the generalization ability and detection effect of the improved model, the best model was subjected to unified inference and performance comparison on a separately reserved test set.

[0037] Table 2 shows the performance comparison results between the YOLOv8 model and the CA-YOLOv8 model. The data indicates that the CA-YOLOv8 model outperforms the YOLOv8 model in most key metrics. Specifically, in the "Comprehensive" category, CA-YOLOv8's precision increased from 91.8% to 96.2%, recall increased from 92.5% to 95.4%, and mean precision increased from 94.0% to 95.2%. Particularly in the "Mature" category, the recall significantly improved from 91.1% to 97.2%, demonstrating the effectiveness of the coordinate attention mechanism in improving the overall model performance.

[0038] A diagram illustrating the comparison of detection results between the YOLOv8 model and the CA-YOLOv8 model, as shown below. Figure 5 As shown. Among them, Figure 5 Part (a) shows the original input image, while parts (b) and (c) show the detection results of YOLOv8 and CA-YOLOv8, respectively. The visualization results clearly show that CA-YOLOv8 is more accurate in locating durian targets in complex scenes, with a significant reduction in missed and false detections, further verifying its advantages in perceiving subtle features and spatial structures.

[0039] Table 2. Performance Comparison Results of YOLOv8 Model and CA-YOLOv8 Model

[0040] Example 2 like Figure 1 As shown in the figure, this embodiment discloses a method for detecting durian maturity based on an improved YOLOv8n and a multimodal large model, which mainly includes the following steps: S1. Obtain a dataset of durian images at different maturity levels from the Roboflow Universe platform, wherein the Roboflow Universe platform is a public image dataset platform that provides rich image resources and convenient data management functions; S2. Randomly divide the dataset into training, validation, and test sets, and perform data augmentation on the training set; S3. Use the LabelImg tool to label the unripe and ripe states of durians. Hereinafter, the unripe state is abbreviated as Unripe and the ripe state is abbreviated as Ripe. The LabelImg tool is an open-source tool for image labeling, which is widely used in target detection tasks in the field of computer vision. S4. Embed coordinate attention mechanism into YOLOv8 model to construct CA-YOLOv8 model. The YOLOv8 model is the next major update of YOLOv5, which was open-sourced by Ultralytics on January 10, 2023. It performs well in image classification, object detection and instance segmentation tasks. S5. Train and optimize the CA-YOLOv8 model using the training set and validation set respectively; S6. Input the detection results of the CA-YOLOv8 model into the multimodal large model HuaNong_Vision to generate a comprehensive analysis report on durian maturity detection. The multimodal large model HuaNong_Vision is a professional multimodal large model for the agricultural field. Its core capability lies in the deep integration of visual information and text knowledge to achieve intelligent cognition and decision support in agricultural scenarios. That is, it supports inputting image information and text information in the agricultural field and outputting a comprehensive analysis report on agricultural scenarios.

[0041] The following is a detailed explanation of each of the above steps: In step S1 above, refer to the detailed description of step S1 in Example 1.

[0042] In step S2 above, a fixed random seed is set before splitting the dataset, and the complete dataset is randomly shuffled. It is then divided into training, validation, and test sets in an 8:1:1 ratio.

[0043] In this embodiment, the implementation steps for training set data augmentation are the same as those for step S2 in embodiment 1.

[0044] In step S3 above, refer to the detailed description of step S3 in Example 1.

[0045] In step S4 above, refer to the detailed description of step S4 in Example 1.

[0046] In step S5 above, the deep learning framework PyTorch 2.0.0, the GPU acceleration library CUDA 11.8, the development platform PyCharm, the programming language Python 3.10.18, and the operating system Ubuntu 18.04 are used; the hardware configuration uses an Intel(R) Xeon(R) Platinum 8255C CPU @2.50GHz processor and an NVIDIA GeForce RTX 4090 graphics card with a 24210MiB.

[0047] In this embodiment, the training parameters of the model are set, and the training parameters and parameter values ​​are shown in Table 3.

[0048] Table 3. Model Training Parameters and Parameter Values

[0049] In this embodiment, the implementation steps for calculating the performance index are the same as those for step S5 in embodiment 1.

[0050] In step S6 above, refer to the detailed description of step S6 in Example 1.

[0051] In this embodiment, the YOLOv5, YOLOv8, and CA-YOLOv8 models were trained for 100 iterations using the training set, respectively. Based on the overall performance of the models on the validation set (mainly referring to the average precision), the best weight file (best.pt) of each YOLOv5, YOLOv8, and CA-YOLOv8 model on the validation set was selected as the final model. To objectively evaluate the generalization ability and detection effect of the improved models, the best models were subjected to unified inference and performance comparison on a separately reserved test set.

[0052] Table 4 shows the performance comparison results of the YOLOv5, YOLOv8, and CA-YOLOv8 models. The data indicates that the CA-YOLOv8 model outperforms both YOLOv5 and YOLOv8 models in most key metrics. Specifically, in the "Comprehensive" category, the mean accuracy of CA-YOLOv8 is 3.3 percentage points higher than that of the YOLOv5 model and 2 percentage points higher than that of the YOLOv8 model, demonstrating the effectiveness of the coordinate attention mechanism in improving the overall performance of the models.

[0053] Table 4. Performance Comparison Results of YOLOv5, YOLOv8, and CA-YOLOv8 Models

[0054] In summary, Examples 1 and 2 disclose a durian maturity detection method based on an improved YOLOv8n and a multimodal large model. By integrating a coordinate attention mechanism into the YOLOv8 model to construct a CA-YOLOv8 model, the method significantly enhances the model's ability to perceive subtle features of durian fruits and model spatial structural relationships. While maintaining high inference efficiency, this method effectively improves the accuracy and robustness of maturity determination in complex environments. Furthermore, by combining the multimodal large model HuaNong_Vision, accurate visual detection results are transformed into a comprehensive and interpretable analysis report. Examples 1 and 2 fully verify the significant advantages of the proposed solution in improving detection accuracy and output information dimensionality, providing an efficient and reliable technical implementation path for intelligent agricultural detection.

[0055] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0056] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.< / height> < / width>

Claims

1. A method for detecting durian maturity based on an improved YOLOv8n and a multimodal large model, characterized in that, The method for detecting durian ripeness includes the following steps: S1. Obtain a dataset of durian images at different maturity levels from the Roboflow Universe platform; S2. Randomly divide the dataset into training, validation, and test sets, and perform data augmentation on the training set; S3. Use the LabelImg tool to label the unripe and ripe states of durians. Hereinafter, the unripe state will be abbreviated as Unripe and the ripe state as Ripe. S4. Embed coordinate attention mechanism into YOLOv8 model to construct CA-YOLOv8 model; S5. Train and optimize the CA-YOLOv8 model using the training set and validation set respectively; S6. Input the detection results of the CA-YOLOv8 model into the multimodal large model HuaNong_Vision to generate a comprehensive analysis report on durian maturity detection.

2. The method for detecting durian maturity based on an improved YOLOv8n and a multimodal large model according to claim 1, characterized in that, The process of step S1 is as follows: S1.

1. Using the Roboflow Universe platform, search using keywords related to durian maturity to obtain a publicly available dataset of durian images containing different maturity stages; S1.2 Filter the acquired images and remove blurry, duplicate, or unclear images.

3. The method for detecting durian maturity based on an improved YOLOv8n and a multimodal large model according to claim 1, characterized in that, The process of step S2 is as follows: S2.1 After randomly shuffling the dataset, divide it into training set, validation set and test set in a ratio of 7:2:1; S2.2 Perform data augmentation on the training set, including flipping, scaling, center cropping, adding noise, and tone transformation.

4. The method for detecting durian maturity based on an improved YOLOv8n and a multimodal large model according to claim 1, characterized in that, The process of step S3 is as follows: S3.1 Using the LabelImg tool, label each individual durian in the image with a rectangular box, classify the green durian as Unripe, and classify the yellow-green, golden yellow, or yellowish-brown durian as Ripe; S3.2 Save the annotation results of each image in the dataset in YOLO format.

5. The method for detecting durian maturity based on an improved YOLOv8n and a multimodal large model according to claim 1, characterized in that, The process of step S4 is as follows: S4.

1. Using the initial feature map output from the backbone network of the YOLOv8 model as input, perform global average pooling along both the width and height spatial dimensions to generate a pair of orientation-aware feature vectors: ， ， in, and These are feature vectors perceived in the width and height directions, respectively. and The width and height dimensions are respectively, F() represents the input feature map, i and j are the loop variables for traversing and summing in the height and width directions, and w and h are the position indices of the vector in the width and height directions, respectively. S4.2, will and The data is then concatenated, followed by channel dimensionality reduction using a 1×1 convolution kernel to reduce the number of channels to 1 / r of the original number. After batch normalization and ReLU activation, intermediate feature maps are generated. ， in, This represents the generated intermediate feature map, where C is the number of channels in the feature map. This represents the channel reduction ratio. () is the ReLU activation function, and BN() is the batch normalization operation. It is a 1×1 convolution kernel. For splicing operations; S4.

3. Split the intermediate feature map f into two feature maps along the spatial dimension, and restore their channel count to 100% using a 1×1 convolution kernel. Then, the sigmoid activation function is used to generate width attention weights and height attention weights: ， ， in, and These are the width attention weights and the height attention weights, respectively. () represents the Sigmoid activation function. and These represent the feature maps after splitting the intermediate feature map f along the width and height dimensions, respectively. and They respectively represent the effects on and A 1×1 convolution kernel; S4.4, Weighting high attention and width attention weight Element-wise multiplication is performed with the initial feature map F to achieve coordinate-aware weighting of the original features, and the enhanced feature map is output. , where F represents the initial feature map, which is short for the input feature map F().

6. The method for detecting durian maturity based on an improved YOLOv8n and a multimodal large model according to claim 1, characterized in that, The process of step S5 is as follows: The CA-YOLOv8 model is trained using the training set, and the training process is monitored using the validation set to prevent overfitting. The best CA-YOLOv8 model is selected based on the performance metrics on the validation set.

7. The method for detecting durian maturity based on an improved YOLOv8n and a multimodal large model according to claim 1, characterized in that, The process of step S6 is as follows: S6.

1. The CA-YOLOv8 model is used to detect the images in the test set. The obtained information on the location, category confidence, and maturity status of durians is used as the input of the multimodal large model HuaNong_Vision. S6.2 The multimodal large model HuaNong_Vision, combined with its internal prior knowledge, performs contextual understanding and reasoning on the detection results, and generates a comprehensive analysis report including maturity score (0-10 points), durian variety, consumption suggestions, storage guidelines and explanations of key judgment criteria.

8. The method for detecting durian maturity based on an improved YOLOv8n and a multimodal large model according to claim 1, characterized in that, The durian maturity detection method also includes a comparative evaluation step of performance indicators and detection results using the YOLOv8 model and the CA-YOLOv8 model, as follows: S5.

1. Train the parameters of the YOLOv8 model and the CA-YOLOv8 model using the training set respectively, and use the validation set to monitor the training process and prevent overfitting. At the same time, select the best model based on the performance indicators on the validation set. S5.2: After training is completed, the performance metrics and detection results of the selected best YOLOv8 model and the best CA-YOLOv8 model are compared and evaluated using the test set. S5.3 Calculate the performance indicators using the following formulas: ， ， ， ， Where Precision represents accuracy, TP represents the number of correctly predicted samples, and TP+FP represents the total number of predicted samples; Recall represents recall, and TP+FN represents the actual number of positive samples. denoted by p, which represents the average precision of the category predictions, and r, which represents the recall rate; mAP represents the mean average precision of all category predictions, and N represents the total number of categories.

Images