Tobacco leaf maturity detection method, device, equipment and medium

Abstract

The invention relates to a tobacco leaf maturity detection method and device, equipment and a medium, and the method comprises the steps: obtaining a to-be-picked tobacco leaf image in a tobacco field environment in response to a tobacco leaf maturity detection instruction; performing target detection on the to-be-picked tobacco leaf image based on a tobacco leaf maturity detection model trained to a convergence state so as to determine candidate frames corresponding to tobacco leaves in the to-be-picked tobacco leaf image and probability prediction of maturity levels; processing the candidate frame by adopting a non-maximum suppression algorithm, and returning the two-dimensional coordinate position and the maturity level corresponding to the tobacco leaves in the to-be-picked tobacco leaf image; the three-dimensional coordinate position of the mature tobacco leaves is determined according to the two-dimensional coordinate position and depth information obtained by a depth camera, and the tobacco leaf harvesting machine picks the mature tobacco leaves according to the three-dimensional coordinate position so as to complete detection of the maturity of the tobacco leaves. According to the method, the detection accuracy of the maturity of the tobacco leaves in a complex environment can be remarkably improved.

Description

Technical Field

[0001] The present application relates to the field of image detection, and in particular to a tobacco leaf maturity detection method, a corresponding device, an electronic device and a computer-readable storage medium. Background Art

[0002] As an important cash crop, tobacco plays an important role in my country's national economy. my country ranks first in the world in terms of tobacco planting area and total output, with the annual planting area maintaining at more than 1,000,000 hectares. In addition, the taxes paid by the tobacco industry have always been an important source of national tax revenue.

[0003] The maturity of tobacco leaves is a key factor affecting the quality of tobacco leaves, and is closely related to the color, aroma and flavor of the tobacco leaves after baking. Harvesting fresh tobacco leaves of appropriate maturity plays a vital role in the production of high-quality tobacco leaves. At present, the maturity identification of tobacco leaves in the field mainly relies on the naked eye observation of tobacco farmers, and the identification quality is uneven, the efficiency is low and the reliability is insufficient. In addition, the existing tobacco leaf harvesting machinery has not yet performed harvesting operations based on the maturity of tobacco leaves. Therefore, it is urgent to carry out research on the automated discrimination of tobacco leaf maturity to ensure the accuracy and objectivity of the discrimination, and at the same time provide a technical basis for intelligent tobacco leaf harvesting machinery based on maturity.

[0004] At present, there are three main methods for the automatic identification of tobacco leaf maturity: based on SPAD value, based on hyperspectral technology and based on computer vision technology. The relative chlorophyll content of tobacco leaves of different maturity is measured by a chlorophyll meter, and on this basis, a chlorophyll meter reading model for determining tobacco leaf maturity is established. However, this method requires the use of a measuring plate to press the leaves, which may cause damage to the tobacco leaves. Near-infrared spectroscopy and hyperspectral technology extract the spectral features of tobacco leaves, analyze the relationship between them and the maturity of tobacco leaves, and establish a tobacco leaf maturity classification model. However, this type of technology has strict requirements on environmental conditions and a long analysis and detection cycle, and is not suitable for real-time identification of fresh tobacco leaves in tobacco fields. In contrast, computer vision technology is more suitable for real-time identification of tobacco leaves in the field.

[0005] There are a lot of studies that use traditional machine learning methods to identify the maturity of tobacco leaves, but these methods require complex feature engineering, involve data dimensionality reduction, and have low classification accuracy, and have difficulty in recognizing images under complex backgrounds. At present, there are few studies that use deep learning models to identify the maturity of fresh tobacco leaves in complex field environments, and most of them are conducted in environments with simpler backgrounds. In addition, most tobacco leaf maturity discrimination algorithms fail to detect tobacco leaves at the same time and cannot provide tobacco leaf location information for tobacco leaf harvesting machinery.

[0006] In the complex field environment, factors such as partial occlusion of leaves, various geometric shape changes, and color and texture complexity caused by illumination changes increase the difficulty of feature extraction and classification. These challenges make the above algorithms face high false detection and missed detection rates in practical applications, limiting their effectiveness and reliability in dynamic and complex environments.

[0007] To sum up, in the existing technology, factors such as partial occlusion of leaves, various geometric shape changes, and color and texture complexity caused by lighting changes in complex field environments increase the difficulty of the feature extraction and classification process, making the algorithm face high false detection rate and missed detection rate in practical applications. The applicant has made corresponding explorations to solve this problem. Summary of the invention

[0008] The purpose of the present application is to solve the above-mentioned problems and to provide a tobacco leaf maturity detection method, a corresponding device, an electronic device and a computer-readable storage medium.

[0009] In order to meet the various objectives of this application, this application adopts the following technical solutions:

[0010] A tobacco leaf maturity detection method proposed to meet one of the purposes of this application comprises:

[0011] Respond to the tobacco leaf maturity detection instruction and obtain the image of the tobacco leaves to be picked in the tobacco field environment;

[0012] Based on the tobacco leaf maturity detection model that has been trained to a convergent state, target detection is performed on the image of tobacco leaves to be picked to determine the candidate boxes corresponding to the tobacco leaves in the image of tobacco leaves to be picked and the probability prediction of the maturity level, wherein the basic network architecture of the tobacco leaf maturity detection model is an improved YOLOv5s model, and the improved YOLOv5s model introduces a CBAM attention mechanism after the last C3 module in the head network of the original YOLOv5s model, introduces a CBAM attention mechanism after the SPPF module, and introduces a WIoU loss function to replace the original CIOU loss function, and updates the optimizer to an Adam optimizer;

[0013] The candidate frame is processed by a non-maximum suppression algorithm, and the corresponding two-dimensional coordinate position and maturity grade of the tobacco leaves in the image of the tobacco leaves to be picked are returned;

[0014] The three-dimensional coordinate position of the mature tobacco leaves is determined according to the two-dimensional coordinate position and the depth information obtained by the depth camera, and the tobacco harvester picks the mature tobacco leaves according to the three-dimensional coordinate position to complete the detection of the maturity of the tobacco leaves.

[0015] Optionally, after the step of acquiring the image of the tobacco leaves to be picked in the tobacco field environment, the following steps are included:

[0016] The labelimg annotation tool is used to annotate the collected tobacco leaves images corresponding to the tobacco leaves with different maturity levels.

[0017] The labeled tobacco leaves to be picked are subjected to random rotation, random cropping, color jitter, Gaussian noise, horizontal flipping, and vertical flipping operations to perform data enhancement on the images of tobacco leaves to be picked at different maturity levels.

[0018] Optionally, the steps of training a tobacco leaf maturity detection model include:

[0019] Acquire a training set of a tobacco leaf maturity detection model, wherein the training set includes a plurality of training samples, and the training samples are constructed by the images of the tobacco leaves to be picked and their corresponding maturity grade labels;

[0020] Inputting the training set into a preset tobacco leaf maturity detection model, calculating the error between the model prediction frame and the true frame, and updating the network parameters according to the error;

[0021] The Adam optimizer is used for back propagation to update the model weights to minimize the WIoU loss function until the tobacco leaf maturity detection model reaches a convergence state, so as to determine the tobacco leaf maturity detection model that has been trained to a convergence state.

[0022] Optionally, the maturity level includes mature or immature.

[0023] A tobacco leaf maturity detection device provided for another purpose of the present application comprises:

[0024] A tobacco leaf image acquisition module is configured to respond to a tobacco leaf maturity detection instruction and acquire an image of tobacco leaves to be picked in a tobacco field environment;

[0025] A target detection module is configured to perform target detection on the image of tobacco leaves to be picked based on a tobacco leaf maturity detection model that has been trained to a convergence state, so as to determine a candidate box corresponding to the tobacco leaves in the image of tobacco leaves to be picked and a probability prediction of the maturity level, wherein the basic network architecture of the tobacco leaf maturity detection model is an improved YOLOv5s model, and the improved YOLOv5s model introduces a CBAM attention mechanism after the last C3 module in the head network of the original YOLOv5s model, introduces a CBAM attention mechanism after the SPPF module, and introduces a WIoU loss function to replace the original CIOU loss function, and updates the optimizer to an Adam optimizer;

[0026] A tobacco leaf maturity determination module is configured to process the candidate frame using a non-maximum suppression algorithm, and return the corresponding two-dimensional coordinate position and maturity level of the tobacco leaves in the image of the tobacco leaves to be picked;

[0027] The mature tobacco leaf picking module is configured to determine the three-dimensional coordinate position of the mature tobacco leaves according to the two-dimensional coordinate position and the depth information obtained by the depth camera. The tobacco leaf harvester picks the mature tobacco leaves according to the three-dimensional coordinate position to complete the detection of the maturity of the tobacco leaves.

[0028] An electronic device provided to meet another purpose of the present application includes a central processing unit and a memory, wherein the central processing unit is used to call and run a computer program stored in the memory to execute the steps of the tobacco leaf maturity detection method described in the present application.

[0029] A computer-readable storage medium is provided to meet another purpose of the present application, which stores a computer program implemented according to the tobacco leaf maturity detection method in the form of computer-readable instructions. When the computer program is called and executed by a computer, the steps included in the corresponding method are executed.

[0030] Compared with the prior art, the present application aims at the factors such as partial occlusion of leaves, various geometric shape changes, and color and texture complexity caused by illumination changes in the complex field environment, which increase the difficulty of feature extraction and classification process, making the algorithm face high false detection rate and missed detection rate in practical application. The present application includes but is not limited to the following beneficial effects:

[0031] First, a tobacco leaf dataset was constructed in a real complex environment: a large number of fresh tobacco leaf images were collected in a real tobacco field environment, including random interferences such as other leaves, tobacco stems, soil, and human hands, and the images were collected under different lighting conditions. By introducing complex background factors such as non-foreground tobacco leaves, the dataset was made closer to the real tobacco field environment, enhancing the robustness and generalization ability of the model in complex environments.

[0032] Second, an improved tobacco leaf detection and maturity discrimination algorithm is proposed:

[0033] (1) Application and optimization of lightweight models: The tobacco maturity detection model of this application adopts the improved lightweight YOLOv5s model as the basic network architecture, sets the depth factor to 0.33 and the width factor to 0.50, and achieves a good balance between detection speed and accuracy. The model parameter volume is only 7.72M, which is suitable for deployment on edge devices to meet the needs of real-time field detection.

[0034] (2) Introducing the Wise-IoU (WIoU) loss function: To address the problem of unclear tobacco leaf borders in complex tobacco fields and low-quality examples in the data set, WIoU is introduced to replace the original CIOU loss function. WIoU uses a dynamic non-monotonic focusing mechanism to evaluate the quality of anchor frames through "outlier degree", which reduces the negative impact of low-quality labels on model training and improves the performance and robustness of the detector.

[0035] (3) Adopting Adam optimizer: The optimizer was changed to the advanced Adam optimizer, which combines the advantages of AdaGrad and RMSProp and can automatically adjust the learning rate to better adapt to the characteristics of the tobacco leaf dataset, thereby improving the convergence speed and training effect of the model.

[0036] (4) Introducing the CBAM attention mechanism: The convolutional block attention module (CBAM) is introduced after the SPPF module and the last C3 module in the Head part to enhance the network's feature extraction capability for fused multi-scale feature maps. CBAM improves the model's attention to important features by weighting the channels and spatial dimensions of the feature map, thus enhancing the prediction accuracy in complex environments.

[0037] Third, the integration of tobacco leaf detection and maturity discrimination is achieved: The model of this application achieves an organic integration in tobacco leaf detection and maturity discrimination. It can not only accurately detect tobacco leaves, but also discriminate their maturity level in real time. By combining the application of depth cameras, the model uses depth information to capture the three-dimensional coordinate position of tobacco leaves in space while obtaining two-dimensional image information of tobacco leaves. This three-dimensional position information is crucial for the automated harvesting of tobacco leaf harvesters.

[0038] Fourth, the traditional tobacco leaf detection method mainly focuses on the presence or absence of tobacco leaves, lacks the acquisition of spatial position information of tobacco leaves, and cannot meet the needs of automated harvesting for precise positioning. This application fills the shortcomings of the existing technology in obtaining position information by integrating deep learning models with depth cameras. Specifically, the model first uses the improved YOLOv5s network to detect tobacco leaves and obtain the two-dimensional position and maturity level of the tobacco leaves in the image; then, the depth information of the corresponding pixel points is obtained through the depth camera, and the three-dimensional spatial coordinates of the tobacco leaves are calculated.

[0039] Fifth, the robustness and accuracy of the model in complex environments are improved: Through multi-faceted model optimization, the model's feature extraction capability and adaptability to low-quality examples are improved to address missed detections, false detections, and incorrect judgments caused by complex factors such as similar background and foreground and lighting changes, thereby significantly improving detection accuracy in complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0041] Figure 1 This is a schematic diagram of the process of the tobacco leaf maturity detection method in the embodiment of the present application;

[0042] Figure 2 The exemplary structure of a tobacco harvester using a tobacco maturity discrimination algorithm in an embodiment of the present application;

[0043] Figure 3 Schematic diagram of the hardware composition and deployment of the tobacco maturity detection system in the embodiment of the present application;

[0044] Figure 4 The exemplary structure of the improved lightweight yolov5s model in the embodiment of the present application;

[0045] Figure 5 A schematic diagram of determining mature tobacco leaves to be picked in an image of tobacco leaves to be picked in an embodiment of the present application;

[0046] Figure 6 This is a principle block diagram of a tobacco leaf maturity detection device in an embodiment of the present application;

[0047] Figure 7 It is a schematic diagram of the structure of the computer device in the embodiment of the present application. DETAILED DESCRIPTION

[0048] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present application.

[0049] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0050] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with those in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0051] It will be understood by those skilled in the art that the "client", "terminal" and "terminal device" used herein include both devices with wireless signal receivers, which are devices with only wireless signal receivers without transmission capabilities, and devices with receiving and transmitting hardware, which are devices with receiving and transmitting hardware capable of two-way communication on a two-way communication link. Such devices may include: cellular or other communication devices such as personal computers, tablet computers, which have single-line displays or multi-line displays or cellular or other communication devices without multi-line displays; PCS (Personal Communications Service, personal communication system), which can combine voice, data processing, fax and / or data communication capabilities; PDA (Personal Digital Assistant, personal digital assistant), which may include a radio frequency receiver, pager, Internet / intranet access, web browser, notepad, calendar and / or GPS (Global Positioning System, global positioning system) receiver; conventional laptop and / or palmtop computers or other devices, which have and / or include a conventional laptop and / or palmtop computer or other device with and / or including a radio frequency receiver. The "client", "terminal" and "terminal device" used herein may be portable, transportable, installed in a vehicle (air, sea and / or land), or suitable for and / or configured to run locally, and / or in a distributed form, at any other location on the earth and / or in space. The "client", "terminal" and "terminal device" used herein may also be a communication terminal, an Internet terminal, a music / video playing terminal, for example, a PDA, a MID (Mobile Internet Device) and / or a mobile phone with a music / video playing function, or a smart TV, a set-top box and other devices.

[0052] The hardware referred to by the names such as "server", "client", and "service node" in this application is essentially an electronic device with the equivalent capabilities of a personal computer. It is a hardware device with the necessary components revealed by the von Neumann principle, such as a central processing unit (including an arithmetic unit and a controller), a memory, an input device, and an output device. The computer program is stored in its memory, and the central processing unit calls the program stored in the external memory into the internal memory for execution, executes the instructions in the program, and interacts with the input and output devices to complete specific functions.

[0053] It should be pointed out that the concept of "server" referred to in this application can also be extended to the case of server clusters. According to the network deployment principle understood by those skilled in the art, the servers should be logically divided. In physical space, these servers can be independent of each other but can be called through interfaces, or integrated into a physical computer or a set of computer clusters. Those skilled in the art should understand this flexibility, and should not use it to restrict the implementation of the network deployment method of this application.

[0054] Unless expressly specified, one or more technical features of the present application can be deployed on a server for implementation and accessed by a client through a remote call to obtain an online service interface provided by the server, or can be directly deployed and run on a client for access.

[0055] The neural network models referenced or may be referenced in this application, unless expressly specified, can be deployed on a remote server and remotely called on the client, or can be deployed and directly called on a client with sufficient device capabilities. In some embodiments, when it runs on the client, its corresponding intelligence can be obtained through transfer learning to reduce the requirements for the client's hardware operating resources and avoid excessive occupation of the client's hardware operating resources.

[0056] Unless explicitly specified, the various data involved in this application can be stored remotely on a server or on a local terminal device, as long as it is suitable for being called by the technical solution of this application.

[0057] Those skilled in the art should be aware that, although the various methods of the present application are described based on the same concept and thus present commonality to each other, unless otherwise specified, these methods can be independently executed. Similarly, for each embodiment disclosed in the present application, they are all proposed based on the same inventive concept, therefore, concepts with the same expression, and concepts that are appropriately changed for convenience despite different expressions, should be understood as equivalent.

[0058] Unless the mutually exclusive relationship between the embodiments to be disclosed in this application is explicitly stated, the relevant technical features involved in each embodiment can be cross-combined to flexibly construct a new embodiment, as long as such combination does not deviate from the creative spirit of this application and can meet the needs of the prior art or solve certain deficiencies in the prior art. Those skilled in the art should be aware of this flexibility.

[0059] See also Figure 1 In one embodiment, the tobacco leaf maturity detection method of the present application comprises:

[0060] Step S10, responding to the tobacco leaf maturity detection instruction, obtaining an image of the tobacco leaves to be picked in the tobacco field environment;

[0061] The tobacco leaf maturity detection system in the tobacco leaf harvester can respond to tobacco leaf maturity detection instructions and obtain images of tobacco leaves to be picked in a tobacco field environment.

[0062] In some embodiments, Figure 2 As shown, by installing a camera (1), a visual system main control (2), and a display screen (3) on a tobacco harvester, the tobacco maturity detection algorithm can be used to obtain tobacco maturity level information in real time on the display screen.

[0063] A feasible hardware deployment solution is proposed for the tobacco maturity detection system: Figure 3 As shown in the figure, a hardware system consisting of a power supply, a depth camera, a Jetson Orin Nano development board and a display screen was built. This solution is easy to use and can configure the model running environment on the mainboard. The optimal weight file obtained through training is used for reasoning. The camera is aimed at the tobacco leaves whose maturity is to be determined. The maturity and position of the target tobacco leaves can be output in real time on the display screen to achieve real-time detection and maturity determination of tobacco leaves in the field, providing a feasible technical solution for practical applications.

[0064] In some embodiments, after the step of acquiring the image of the tobacco leaves to be picked in the tobacco field environment, the following steps are included:

[0065] Step S101, using labelimg labeling tool to label the collected images of tobacco leaves to be picked corresponding to tobacco leaves to be picked of different maturity levels;

[0066] Step S102, performing random rotation, random cropping, color jittering, Gaussian noise, horizontal flipping, and vertical flipping operations on the labeled tobacco leaves to be picked, so as to perform data enhancement on the images of tobacco leaves to be picked of different maturity levels.

[0067] Specifically, a large number of images of tobacco leaves to be picked with different maturity levels are collected in a real tobacco field environment. The target tobacco leaves occupy the main part of the picture. The picture also contains random interferences existing in the real environment of the tobacco field, such as other leaves, tobacco stems, soil or human hands. The collection environment includes different lighting conditions, which makes the dataset as close as possible to the real tobacco field environment and ensures the diversity of the dataset, so as to improve the robustness of the trained model.

[0068] Under the guidance of professionals, the Labelimg annotation tool was used to manually annotate the picked tobacco leaves, select the target tobacco leaves and annotate the maturity level, and obtain a label file in txt format. The original images were randomly divided into test set, validation set and test set in a ratio of 8:1:1, and then the images of the test set, validation set and test set were data enhanced respectively. In order to make the model have better generalization performance and be able to better judge the maturity of tobacco leaves in various environments, data enhancement methods were used to expand the images. Random image rotation, random image flipping, random brightness adjustment and random noise were used to expand the data set 5 times.

[0069] Step S20, performing target detection on the image of tobacco leaves to be picked based on the tobacco leaf maturity detection model that has been trained to a convergent state, so as to determine the candidate boxes corresponding to the tobacco leaves in the image of tobacco leaves to be picked and the probability prediction of the maturity level, wherein the basic network architecture of the tobacco leaf maturity detection model is an improved YOLOv5s model, and the improved YOLOv5s model introduces a CBAM attention mechanism after the last C3 module in the head network of the original YOLOv5s model, introduces a CBAM attention mechanism after the SPPF module, and introduces a WIoU loss function to replace the original CIOU loss function, and updates the optimizer to an Adam optimizer;

[0070] After obtaining the image of the tobacco leaves to be picked in the tobacco field environment, target detection is performed on the image of the tobacco leaves to be picked based on the tobacco leaf maturity detection model that has been trained to a convergent state to determine the candidate boxes corresponding to the tobacco leaves in the image of the tobacco leaves to be picked and the probability prediction of the maturity level. For details, please refer to Figure 4 The tobacco maturity detection model of the present application adopts an improved lightweight YOLOv5s model as the basic network architecture. The improved YOLOv5s model introduces a CBAM attention mechanism after the last C3 module in the head network of the original YOLOv5s model, introduces a CBAM attention mechanism after the SPPF module, and introduces a WIoU loss function to replace the original CIOU loss function. The optimizer is updated to an Adam optimizer with a depth factor of 0.33 and a width factor of 0.50. It has a good balance between detection speed and accuracy and is suitable for deployment on edge devices.

[0071] The CBAM (Convolutional Block Attention Module) attention mechanism is introduced after the SPPF module in the head network of the original YOLOv5s model to enhance the feature extraction capability of fused multi-scale feature maps;

[0072] The CBAM (Convolutional Block Attention Module) attention mechanism is also introduced after the last C3 module in the head network (head) part to enhance the feature extraction capability of the final 1024×20×20 feature map, so as to build an improved yolov5s model and improve the accuracy of model prediction.

[0073] Furthermore, the borders of tobacco leaves collected in the complex environment of tobacco fields are not clear, resulting in low-quality examples in the dataset. Wise-IoU (WIoU) is introduced to replace the original CIOU. Its dynamic non-monotonic focusing mechanism uses "outliers" instead of IoU to evaluate the quality of anchor boxes, reducing the harmful gradients of low-quality labels during model training and improving the performance of the detector. The optimizer is changed to the advanced Adam optimizer, which combines the advantages of the two optimization algorithms AdaGrad and RMSProp, can automatically adjust the learning rate, and can better adapt to the tobacco dataset.

[0074] The model is implemented using the Pytorch deep learning framework. The training batch size is set to 300. GPU is used for training. After training, the optimal weight pt file is saved.

[0075] Step S30, using a non-maximum suppression algorithm to process the candidate frame, and returning the corresponding two-dimensional coordinate position and maturity grade of the tobacco leaves in the image of the tobacco leaves to be picked;

[0076] Based on the tobacco leaf maturity detection model that has been trained to a convergent state, target detection is performed on the image of tobacco leaves to be picked to determine the candidate frames corresponding to the tobacco leaves in the image of tobacco leaves to be picked and the probability prediction of the maturity level. Then, a non-maximum suppression algorithm is used to process the candidate frames, and the two-dimensional coordinate position corresponding to the tobacco leaves in the image of tobacco leaves to be picked and the maturity level are returned, wherein the maturity level includes mature or unmature.

[0077] In some embodiments, the step of training a tobacco leaf maturity detection model comprises:

[0078] Step S301, obtaining a training set of a tobacco leaf maturity detection model, wherein the training set includes a plurality of training samples, and the training samples are constructed by the images of the tobacco leaves to be picked and their corresponding maturity grade labels;

[0079] Step S302: input the training set into a preset tobacco leaf maturity detection model, calculate the error between the model prediction frame and the real frame, and update the network parameters according to the error;

[0080] Step S303: Use the Adam optimizer to perform back propagation and update the model weights to minimize the WIoU loss function until the tobacco leaf maturity detection model reaches a convergence state, so as to determine the tobacco leaf maturity detection model that has been trained to a convergence state.

[0081] Step S40, determining the three-dimensional coordinate position of the mature tobacco leaves according to the two-dimensional coordinate position and the depth information obtained by the depth camera, and the tobacco harvester picks the mature tobacco leaves according to the three-dimensional coordinate position to complete the detection of tobacco maturity.

[0082] The candidate frame is processed by a non-maximum suppression algorithm, and the corresponding two-dimensional coordinate position and maturity level of the tobacco leaves in the image of the tobacco leaves to be picked are returned. The three-dimensional coordinate position of the mature tobacco leaves is determined according to the two-dimensional coordinate position and the depth information obtained by the depth camera. The tobacco harvester picks the mature tobacco leaves according to the three-dimensional coordinate position to complete the detection of the maturity of the tobacco leaves.

[0083] Specifically, the model of this application achieves an organic integration in tobacco leaf detection and maturity discrimination, which can not only accurately detect tobacco leaves, but also discriminate their maturity level in real time. By combining the application of depth cameras, the model uses depth information to capture the three-dimensional coordinate position of tobacco leaves in space while obtaining two-dimensional image information of tobacco leaves. This three-dimensional position information is crucial for the automated harvesting of tobacco leaf harvesters.

[0084] Traditional tobacco leaf detection methods mainly focus on the presence or absence of tobacco leaves, lack the acquisition of spatial position information of tobacco leaves, and cannot meet the needs of automated harvesting for precise positioning. This application fills the gap in the acquisition of position information in the existing technology by integrating deep learning models with depth cameras. Specifically, the model first uses the improved YOLOv5s network to detect tobacco leaves and obtain the two-dimensional position and maturity level of the tobacco leaves in the image; then, the depth information of the corresponding pixel points is obtained through the depth camera, and the three-dimensional spatial coordinates of the tobacco leaves are calculated.

[0085] In some embodiments, see Figure 5The effectiveness of the model was verified by experimental data: the average accuracy (mAP@0.5) of the improved model reached 85.1% under the premise that the number of parameters remained basically unchanged, which was 3.8% higher than the original model. The maturity discrimination accuracy reached 94.3%, which was able to accurately detect and distinguish the maturity level of tobacco leaves. The experimental results proved the effectiveness and superiority of the proposed method in complex tobacco field environments.

[0086] Compared with the prior art, the present application aims at the factors such as partial occlusion of leaves, various geometric shape changes, and color and texture complexity caused by illumination changes in the complex field environment, which increase the difficulty of feature extraction and classification process, making the algorithm face high false detection rate and missed detection rate in practical application. The present application includes but is not limited to the following beneficial effects:

[0087] First, a tobacco leaf dataset was constructed in a real complex environment: a large number of fresh tobacco leaf images were collected in a real tobacco field environment, including random interferences such as other leaves, tobacco stems, soil, and human hands, and the images were collected under different lighting conditions. By introducing complex background factors such as non-foreground tobacco leaves, the dataset was made closer to the real tobacco field environment, enhancing the robustness and generalization ability of the model in complex environments.

[0088] Second, an improved tobacco leaf detection and maturity discrimination algorithm is proposed:

[0089] (1) Application and optimization of lightweight models: The tobacco maturity detection model of this application adopts the improved lightweight YOLOv5s model as the basic network architecture, sets the depth factor to 0.33 and the width factor to 0.50, and achieves a good balance between detection speed and accuracy. The model parameter volume is only 7.72M, which is suitable for deployment on edge devices to meet the needs of real-time field detection.

[0090] (2) Introducing the Wise-IoU (WIoU) loss function: To address the problem of unclear tobacco leaf borders in complex tobacco fields and low-quality examples in the data set, WIoU is introduced to replace the original CIOU loss function. WIoU uses a dynamic non-monotonic focusing mechanism to evaluate the quality of anchor frames through "outlier degree", which reduces the negative impact of low-quality labels on model training and improves the performance and robustness of the detector.

[0091] (3) Adopting Adam optimizer: The optimizer was changed to the advanced Adam optimizer, which combines the advantages of AdaGrad and RMSProp and can automatically adjust the learning rate to better adapt to the characteristics of the tobacco leaf dataset, thereby improving the convergence speed and training effect of the model.

[0092] (4) Introducing the CBAM attention mechanism: The convolutional block attention module (CBAM) is introduced after the SPPF module and the last C3 module in the Head part to enhance the network's feature extraction capability for fused multi-scale feature maps. CBAM improves the model's attention to important features by weighting the channels and spatial dimensions of the feature map, thus enhancing the prediction accuracy in complex environments.

[0093] Third, the integration of tobacco leaf detection and maturity discrimination is achieved: The model of this application achieves an organic integration in tobacco leaf detection and maturity discrimination. It can not only accurately detect tobacco leaves, but also discriminate their maturity level in real time. By combining the application of depth cameras, the model uses depth information to capture the three-dimensional coordinate position of tobacco leaves in space while obtaining two-dimensional image information of tobacco leaves. This three-dimensional position information is crucial for the automated harvesting of tobacco leaf harvesters.

[0094] Fourth, the traditional tobacco leaf detection method mainly focuses on the presence or absence of tobacco leaves, lacks the acquisition of spatial position information of tobacco leaves, and cannot meet the needs of automated harvesting for precise positioning. This application fills the shortcomings of the existing technology in obtaining position information by integrating deep learning models with depth cameras. Specifically, the model first uses the improved YOLOv5s network to detect tobacco leaves and obtain the two-dimensional position and maturity level of the tobacco leaves in the image; then, the depth information of the corresponding pixel points is obtained through the depth camera, and the three-dimensional spatial coordinates of the tobacco leaves are calculated.

[0095] Fifth, the robustness and accuracy of the model in complex environments are improved: Through multi-faceted model optimization, the model's feature extraction capability and adaptability to low-quality examples are improved to address missed detections, false detections, and incorrect judgments caused by complex factors such as similar background and foreground and lighting changes, thereby significantly improving detection accuracy in complex environments.

[0096] See also Figure 6, a tobacco leaf maturity detection device provided to meet one of the purposes of the present application, includes a tobacco leaf image acquisition module 1100, a target detection module 1200, a tobacco leaf maturity determination module 1300 and a mature tobacco leaf picking module 1400. The tobacco leaf image acquisition module 1100 is configured to respond to a tobacco leaf maturity detection instruction and acquire an image of tobacco leaves to be picked in a tobacco field environment; the target detection module 1200 is configured to perform target detection on the image of tobacco leaves to be picked based on a tobacco leaf maturity detection model that has been trained to a convergence state, so as to determine the candidate boxes corresponding to the tobacco leaves in the image of tobacco leaves to be picked and the probability prediction of the maturity level, wherein the basic network architecture of the tobacco leaf maturity detection model is an improved YOLOv5s model, and the improved YOLOv5s model introduces a CBAM attention mechanism after the last C3 module in the head network of the original YOLOv5s model, The CBAM attention mechanism is introduced after the SPPF module, and the WIoU loss function is introduced to replace the original CIOU loss function, and the optimizer is updated to the Adam optimizer; the tobacco leaf maturity determination module 1300 is configured to use a non-maximum suppression algorithm to process the candidate frame, and return the corresponding two-dimensional coordinate position and maturity level of the tobacco leaves in the image of the tobacco leaves to be picked; the mature tobacco leaf picking module 1400 is configured to determine the three-dimensional coordinate position of the mature tobacco leaves according to the two-dimensional coordinate position and the depth information obtained by the depth camera, and the tobacco leaf harvester picks the mature tobacco leaves according to the three-dimensional coordinate position to complete the detection of the maturity of the tobacco leaves.

[0097] Based on any embodiment of this application, please refer to Figure 7 Another embodiment of the present application further provides an electronic device, which can be implemented by a computer device, such as Figure 7 As shown, a schematic diagram of the internal structure of a computer device. The computer device includes a processor, a computer-readable storage medium, a memory, and a network interface connected via a system bus. Among them, the computer-readable storage medium of the computer device stores an operating system, a database, and computer-readable instructions. The database may store a control information sequence. When the computer-readable instructions are executed by the processor, the processor can implement a tobacco leaf maturity detection method. The processor of the computer device is used to provide computing and control capabilities to support the operation of the entire computer device. The memory of the computer device may store computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor can execute the tobacco leaf maturity detection method of the present application. The network interface of the computer device is used to connect and communicate with a terminal. Those skilled in the art can understand that Figure 7The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0098] In this embodiment, the processor is used to execute Figure 6 The memory stores the program code and various data required to execute the above modules or submodules. The network interface is used to transmit data between user terminals or servers. The memory in this embodiment stores the program code and data required to execute all modules / submodules in the tobacco maturity detection device of the present application, and the server can call the program code and data of the server to execute the functions of all submodules.

[0099] The present application also provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the tobacco leaf maturity detection method described in any embodiment of the present application.

[0100] The present application also provides a computer program product, including a computer program / instruction, which, when executed by one or more processors, implements the steps of the tobacco leaf maturity detection method described in any embodiment of the present application.

[0101] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments of the present application can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a computer-readable storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0102] The above is only a partial implementation method of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

[0103] In summary, this application improves the robustness and accuracy of the model in complex environments. Through multi-faceted model optimization, it improves the model's feature extraction capability and adaptability to low-quality examples, targeting missed detections, false detections, and incorrect discrimination problems caused by complex factors such as similar background and foreground, and lighting changes, thereby significantly improving detection accuracy in complex environments.

Claims

1. A method for detecting tobacco leaf maturity, characterized in that: include: Respond to the tobacco leaf maturity detection instruction and obtain the image of the tobacco leaves to be picked in the tobacco field environment; Based on the tobacco leaf maturity detection model that has been trained to a convergent state, target detection is performed on the image of tobacco leaves to be picked to determine the candidate boxes corresponding to the tobacco leaves in the image of tobacco leaves to be picked and the probability prediction of the maturity level, wherein the basic network architecture of the tobacco leaf maturity detection model is an improved YOLOv5s model, and the improved YOLOv5s model introduces a CBAM attention mechanism after the last C3 module in the head network of the original YOLOv5s model, introduces a CBAM attention mechanism after the SPPF module, and introduces a WIoU loss function to replace the original CIOU loss function, and updates the optimizer to an Adam optimizer; The candidate frame is processed by a non-maximum suppression algorithm, and the corresponding two-dimensional coordinate position and maturity grade of the tobacco leaves in the image of the tobacco leaves to be picked are returned; The three-dimensional coordinate position of the mature tobacco leaves is determined according to the two-dimensional coordinate position and the depth information obtained by the depth camera, and the tobacco harvester picks the mature tobacco leaves according to the three-dimensional coordinate position to complete the detection of the maturity of the tobacco leaves.

2. The tobacco leaf maturity detection method according to claim 1, characterized in that: After the step of obtaining the image of the tobacco leaves to be picked in the tobacco field environment, it includes: The labelimg annotation tool is used to annotate the collected tobacco leaves images corresponding to the tobacco leaves with different maturity levels. The labeled tobacco leaves to be picked are subjected to random rotation, random cropping, color jittering, Gaussian noise, horizontal flipping, and vertical flipping operations to perform data enhancement on the images of tobacco leaves to be picked at different maturity levels.

3. The tobacco leaf maturity detection method according to claim 1, characterized in that: The steps to train the tobacco leaf maturity detection model include: Acquire a training set of a tobacco leaf maturity detection model, wherein the training set includes a plurality of training samples, and the training samples are constructed by the images of the tobacco leaves to be picked and their corresponding maturity grade labels; Inputting the training set into a preset tobacco leaf maturity detection model, calculating the error between the model prediction frame and the true frame, and updating the network parameters according to the error; The Adam optimizer is used for back propagation to update the model weights to minimize the WIoU loss function until the tobacco leaf maturity detection model reaches a convergence state, so as to determine the tobacco leaf maturity detection model that has been trained to a convergence state.

4. The tobacco leaf maturity detection method according to claim 1, characterized in that: The maturity levels include mature or unripe.

5. A tobacco leaf maturity detection device, characterized in that: include: A tobacco leaf image acquisition module is configured to respond to a tobacco leaf maturity detection instruction and acquire an image of tobacco leaves to be picked in a tobacco field environment; A target detection module is configured to perform target detection on the image of tobacco leaves to be picked based on a tobacco leaf maturity detection model that has been trained to a convergence state, so as to determine a candidate box corresponding to the tobacco leaves in the image of tobacco leaves to be picked and a probability prediction of the maturity level, wherein the basic network architecture of the tobacco leaf maturity detection model is an improved YOLOv5s model, and the improved YOLOv5s model introduces a CBAM attention mechanism after the last C3 module in the head network of the original YOLOv5s model, introduces a CBAM attention mechanism after the SPPF module, and introduces a WIoU loss function to replace the original CIOU loss function, and updates the optimizer to an Adam optimizer; A tobacco leaf maturity determination module is configured to process the candidate frame using a non-maximum suppression algorithm, and return the corresponding two-dimensional coordinate position and maturity level of the tobacco leaves in the image of the tobacco leaves to be picked; The mature tobacco leaf picking module is configured to determine the three-dimensional coordinate position of the mature tobacco leaves according to the two-dimensional coordinate position and the depth information obtained by the depth camera. The tobacco leaf harvester picks the mature tobacco leaves according to the three-dimensional coordinate position to complete the detection of the maturity of the tobacco leaves.

6. An electronic device, comprising a central processing unit and a memory, characterized in that: The central processing unit is used to call and run the computer program stored in the memory to execute the steps of the method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that: It stores a computer program implemented according to the method described in any one of claims 1 to 4 in the form of computer-readable instructions, and when the computer program is called and executed by a computer, the steps included in the corresponding method are executed.

Images