Tea leaf green-making degree detection method and device based on multiple modes, equipment and medium

Abstract

The invention relates to the technical field of tea making, in particular to a tea leaf fine manipulation degree detection method, device and equipment based on multiple modalities and a medium, and the method comprises the steps that image data, temperature and humidity data and weight data of tea leaves in the fine manipulation process are obtained through corresponding sensors; obtaining a first classification result of the green-making leaves according to the image data, and obtaining a second classification result of the green-making leaves according to the temperature and humidity data and the weight data; and finally, summarizing the first classification result and the second classification result to determine the green-making degree of the tea leaves. Therefore, the problems of single judgment condition, long consumed time, complex operation, low accuracy and the like when the tea is detected based on an image processing method in the related technology are solved.

Description

Technical Field

[0001] The present application relates to the technical field of tea making, and in particular to a method, device, equipment and medium for detecting the degree of greenness of tea leaves based on multimodality. Background Art

[0002] Tea has a long history in my country. Its production process is mainly divided into a series of processes such as withering, making green tea, frying green tea, rolling, and baking. Among them, making green tea is the key process in the primary processing of tea, which has a key influence on the formation of tea aroma and taste and other quality characteristics. During the process of making green tea, the leaves will continue to rub against each other as the barrel rotates and the temperature in the barrel continues to rise, causing the active substances of tea polyphenols in the edges of the leaves to be partially oxidized into bright yellow theaflavins, which are further oxidized into reddish-brown thearubigins. Therefore, the next step of making green tea can be judged by the color change of the leaf edges. In the process of mass production and processing of tea, tea makers usually rely on experience to judge the degree of making green tea, which not only requires a lot of time and energy, but also due to the influence of subjective factors, it is difficult to ensure the consistency of the quality of different batches of tea.

[0003] With the development of computer vision and artificial intelligence technology, many image processing-based methods have emerged to detect the greening of tea leaves. The greening process needs to be carried out continuously to ensure the normal fermentation of tea leaves. Methods such as using a colorimeter, establishing an RGB model, and utilizing a color histogram of tea images require the tea leaves to be taken out for separate processing. This not only requires the collection of large-scale data, but also has weak model generalization capabilities, and requires a lot of computing time and complex operations. In addition, the judgment conditions are single and are not suitable for use in actual production and life, and cannot meet the real-time and accuracy of detection. Summary of the invention

[0004] The present application provides a method, device, equipment and medium for detecting the degree of greenness of tea leaves based on multimodality, so as to solve the problems in the related art of detecting tea leaves based on image processing, which has a single judgment condition, a long time consumption, complex operation and low accuracy.

[0005] The first aspect of the present application provides a method for detecting the degree of greening of tea leaves based on multimodality, comprising the following steps: acquiring image data, temperature and humidity data, and weight data of the tea leaves during the greening process; identifying a first classification result of greening leaves based on the image data, and identifying a second classification result of greening leaves based on the temperature and humidity data and weight data; and determining the degree of greening of the tea leaves based on the first classification result and the second classification result.

[0006] Optionally, identifying a first classification result of green leaves based on image data includes: inputting the image data into a first network model, the first network model performing tea semantic segmentation on the image data, and outputting a first classification result of green leaves in the segmented image data, wherein the first network model is based on MobileNetV3-Large, and includes an efficient multi-scale attention module and a lightweight spatial pyramid pooling solution.

[0007] Optionally, the multi-scale attention module includes: multiple convolutional subnetworks and grouping operations to ensure the ability to extract features in global spatial information, local spatial features, and channel dimensions. The lightweight spatial pyramid pooling solution adopts a three-branch structure to extract feature maps at different sampling scales, so that the model integrates the advantages of deep and shallow features when fusing features, effectively improving the accuracy of segmentation tasks.

[0008] Optionally, the first network model outputs the segmented mask map as the first classification result of the green leaves, including: obtaining feature information of the target at different scales based on the multi-scale attention module, so that the target is easier to detect and focus on; fusing the feature maps of different network levels based on the lightweight spatial pyramid pooling solution to ensure the balance of features in local and global information; generating the first classification result according to the ratio of the amount of feature pixels in the mask map obtained by the decoding head fusion processing to the area of ​​the entire picture.

[0009] Optionally, a second classification result of tea leaves for greening is identified based on temperature and humidity data and weight data, including: inputting the temperature and humidity data and weight data into a second network model, the second network model outputting the weight loss rate of tea leaves during the greening process; and generating a second classification result based on the weight loss rate of tea leaves.

[0010] Optionally, the second network model includes: an input layer, a hidden layer and an output layer, wherein a first connection weight is set between the input layer and the hidden layer, a second connection weight is set between the hidden layer and the output layer, and an input gate, an output gate and a forget gate are set inside the hidden layer.

[0011] Optionally, before inputting the image data into the first network model, it also includes: obtaining an image dataset of tea leaves, wherein the image dataset is divided into a training set, a test set and a validation set; using the training set to perform classification training on the first network model, and using the test set and the validation set to test and verify the trained first network model to verify whether the model structure design achieves the expected effect.

[0012] The second aspect of the present application provides a multi-modal tea leaf ripening degree detection device, including: an acquisition module, used to acquire image data, temperature and humidity data, and weight data of tea leaves during the ripening process; an identification module, used to identify a first classification result of the ripened leaves according to the image data, and to identify a second classification result of the ripened leaves according to the temperature and humidity data and the weight data; a determination module, used to determine the ripening degree of the tea leaves according to the first classification result and the second classification result.

[0013] Optionally, the recognition module is further used to input the image data into a first network model, the first network model performs tea semantic segmentation on the image data, and outputs a first classification result of green leaves in the segmented image data, wherein the first network model includes an efficient multi-scale attention module and a lightweight spatial pyramid pooling solution.

[0014] Optionally, the multi-scale attention module includes: multiple convolutional subnetworks and grouping operations to ensure the ability to extract features in global spatial information, local spatial features, and channel dimensions. The lightweight spatial pyramid pooling solution adopts a three-branch structure to extract feature maps at different sampling scales, so that the model integrates the advantages of deep and shallow features when fusing features, effectively improving the accuracy of segmentation tasks.

[0015] Optionally, identifying a first classification result of the tea-making process according to the image data includes: inputting the image data into a first network model, the first network model performing tea semantic segmentation on the image data, and outputting a first classification result of the tea-making process in the segmented image data, wherein the first network model is based on MobileNetV3-Large, and includes an efficient multi-scale attention module and a lightweight spatial pyramid pooling solution.

[0016] Optionally, a second classification result of tea leaves for greening is identified based on temperature and humidity data and weight data, including: inputting the temperature and humidity data and weight data into a second network model, the second network model outputting the weight loss rate of tea leaves during the greening process; and generating a second classification result based on the weight loss rate of tea leaves.

[0017] Optionally, the second network model includes: an input layer, a hidden layer and an output layer, wherein a first connection weight is set between the input layer and the hidden layer, a second connection weight is set between the hidden layer and the output layer, and an input gate, an output gate and a forget gate are set inside the hidden layer.

[0018] Optionally, the multimodal tea leaf greenness detection device also includes a training module, which is used to obtain a tea leaf greenness image data set before inputting the segmented image data into the first network model, wherein the image data is divided into a training set, a test set and a verification set; the training set is used to perform classification training on the first network model, and the test set and the verification set are used to test and verify the trained first network model until the preset conditions are met and the training is stopped.

[0019] The third aspect of the present application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement a multi-modal method for detecting the degree of greenness of tea leaves as described in the above embodiment.

[0020] The fourth aspect of the present application provides a computer-readable storage medium having a computer program or instruction stored thereon. When the computer program or instruction is executed, it is used to implement the multi-modal tea leaf greenness detection method as described in the above embodiment.

[0021] Therefore, this application has at least the following beneficial effects:

[0022] The embodiment of the present application combines the image data of the tea leaves during the greening process with the temperature, humidity and weight changes, obtains the first classification result through the image data of the tea leaves, obtains the second classification result through the temperature, humidity and weight characteristics during the greening process, and performs decision-making-level fusion on the classification results to judge the overall greening degree of the tea leaves. Based on the semantic information of the greening leaves, it can achieve a more accurate greening degree classification by combining the temperature, humidity and weight characteristics during the greening process, thereby improving the accuracy of the overall detection. Thus, the image processing-based method in the related art is used to detect tea leaves, which has a single judgment condition, a long time, complex operation, and low accuracy.

[0023] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0025] Figure 1 A flowchart of a method for detecting the degree of greenness of tea leaves based on multimodality provided according to an embodiment of the present application;

[0026] Figure 2 This is an example diagram of temperature and humidity data changes provided according to an embodiment of the present application;

[0027] Figure 3This is an example diagram of the change of original weight data provided according to an embodiment of the present application;

[0028] Figure 4 This is an example diagram of processed weight data provided according to an embodiment of the present application;

[0029] Figure 5 A schematic diagram of a multi-scale attention module provided according to an embodiment of the present application;

[0030] Figure 6 A schematic diagram of a lightweight space pyramid pool for dissolving a dock according to an embodiment of the present application;

[0031] Figure 7 A diagram showing the segmentation effects of each network on a test set provided according to an embodiment of the present application;

[0032] Figure 8 An example diagram of a long short-term memory network provided according to an embodiment of the present application;

[0033] Fig. 9 A green leaf state prediction diagram provided according to an embodiment of the present application;

[0034] Fig.10 A decision-level fusion structure diagram provided according to an embodiment of the present application;

[0035] Fig.11 This is an example diagram of a method for detecting the degree of greenness of tea leaves based on multimodality according to an embodiment of the present application;

[0036] Fig.12 is a block diagram of a tea leaf greenness detection device based on multi-modality according to an embodiment of the present application;

[0037] Fig.13 Schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0038] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0039] The following describes the method, device, equipment and storage medium for detecting the degree of greening of tea leaves based on multimodality according to the embodiments of the present application with reference to the accompanying drawings. In view of the problems mentioned in the above background technology, the present application provides a method for detecting the degree of greening of tea leaves based on multimodality. In this method, the image data and the temperature, humidity and weight changes during the greening process of tea leaves are combined, and the first classification result is obtained through the image data of the tea leaves, and the second classification result is obtained through the temperature, humidity and weight characteristics during the greening process. The classification results are fused at the decision level to determine the degree of greening of the tea leaves. Based on the semantic information of the greening leaves, a more accurate classification of the degree of greening can be achieved by combining the temperature, humidity and weight characteristics during the greening process, thereby improving the accuracy of the overall detection. Thus, the problems of detecting tea leaves based on image processing in the related art, which have a single judgment condition, and exist in the form of long time, complex operation and low accuracy, are solved.

[0040] Specifically, Figure 1 A schematic flow chart of a method for detecting the degree of greenness of tea leaves based on multimodality provided in an embodiment of the present application.

[0041] like Figure 1 As shown, the multimodal tea greenness detection method includes the following steps:

[0042] In step S101, image data, temperature and humidity data, and weight data of tea leaves during the greening process are obtained.

[0043] It is understandable that in the embodiment of the present application, a visual device can be installed on the tea-making machine to capture images of tea leaves, such as a visual device including components such as a camera, a NodeMCU wireless transmission device, a fill light, and a mobile power supply. Since the tea-making environment is usually dark and closed, a fill light is used to provide the necessary lighting for the camera to ensure that a clear image of the tea leaves is captured. The image captured by the camera will be transmitted to the Raspberry Pi 4B controller in real time for storage and preliminary processing. In order to facilitate subsequent analysis, the image size can be adjusted to a uniform size, such as 512×512 pixels in the embodiment of the present application.

[0044] The embodiment of the present application can use the temperature and humidity sensor installed to monitor and record the temperature and humidity changes in the entire greening process. The temperature and humidity data are as follows: Figure 2 As shown, the collected data can be sent to the database in real time for storage via wireless means.

[0045] In the embodiment of the present application, single-point weighing sensors can be installed under the four wheels at the bottom of the tea-making machine to form an overhead structure with the base to measure the overall weight change of the tea leaves and their containers. Through this arrangement, the overall weight of the drum of the tea-making machine during the tea-making process can be obtained. Since the supporting parts of the tea-making barrel rely on four wheels respectively, and after the weight sensor is installed, its spring-like structure will vibrate slightly due to the change in the weight of the tea-making machine, especially after the tea-making machine rotates. Although it will not affect the normal operation of the machine, its output data will output abnormal data when the tea-making machine rotates. Therefore, the embodiment of the present application can pre-process the original weight data. First, the threshold obtained by Z standardization is selected. When the standardized score exceeds the threshold, it is abnormal. Secondly, in order to ensure the continuity in time series, the arithmetic difference method is used to replace the abnormal data. For data missing caused by data transmission failure, linear interpolation is used to fill it. In the actual execution process, the weight sensor outputs data once every 1 minute, and the tea-making time is 557 minutes. The data before preprocessing is shown in the figure below. Figure 3 shown. Figure 3 The abnormal data are all caused by the rotation of the green barrel during the green making process. Therefore, the embodiment of the present application can pre-process the data. The weight data after pre-processing is as follows: Figure 4 shown.

[0046] In step S102, a first classification result of green leaves is identified based on the image data, and a second classification result of green leaves is identified based on the temperature and humidity data and the weight data.

[0047] In one embodiment of the present application, a first classification result of green leaves is identified based on image data, including: inputting the image data into a first network model, the first network model performing tea semantic segmentation on the image data, and outputting the first classification result of green leaves in the segmented image data, wherein the first network model is based on MobileNetV3-Large, and includes an efficient multi-scale attention module and a lightweight spatial pyramid pooling solution.

[0048] Among them, the multi-scale attention module includes: multiple convolutional sub-networks and grouping operations to ensure the ability to extract features in global spatial information, local spatial features and channel dimensions. The lightweight spatial pyramid pooling solution adopts a three-branch structure to extract feature maps at different sampling ratios, so that the model integrates the advantages of deep and shallow features at the same time when fusing features, effectively improving the accuracy of segmentation tasks.

[0049] The EMCA (Efficient Multi-scale Channel Attention) of the embodiment of the present application has a structure as follows Figure 5As shown in the figure, the use of multi-scale parallel sub-networks in EMCA not only avoids more sequential processing of channels and increased channel depth, but also uses multiple branches in the parallel sub-network to reduce the information loss caused by global average pooling, while avoiding the problems of excessive local weights and large number of parameters caused by fully connected operations, thereby improving the feature extraction capability.

[0050] As a specific example, in the EMCA structure, first, the channels of the input feature map will be grouped to reduce the number of channels and the amount of calculation in each group. The grouping operation allows the feature channels in each group to be processed in parallel, which improves the parallelism of the model and allows the model to better learn and capture feature information of different scales and semantics. Secondly, three convolution kernels are placed in parallel in three sub-networks in each group. The three sub-networks use one 1×1 convolution kernel and two 3×3 convolution kernels respectively. The first parallel sub-network uses a 1×1 convolution kernel and adopts the same processing method as the CA attention module. It uses the global average pooling operation to process cross-channel information and embeds the spatial position information into the channel attention map to enhance feature aggregation. At the same time, in order to avoid the loss of channel information caused by feature grouping and multi-scale structure, the second 3×3 sub-network is used to establish short-term and long-range dependencies. The third 3×3 branch is responsible for capturing the interaction between channels at the bottom branch, increasing the dependency between the channel dimension and the spatial dimension. Finally, the output feature maps of the two parallel sub-networks are fused through the cross-space learning method to improve the module's ability to extract small targets and scattered features.

[0051] Secondly, the lightweight space pyramid pool decontamination terminal of the embodiment of the present application adopts a three-branch structure (LRASPPHead), such as Figure 6 As shown in the figure, the 1 / 2, 1 / 4, and 1 / 32 downsampled feature maps of the Backbone are extracted for processing respectively. Compared with the LRASPP decoding head, deepening the downsampling multiple can obtain more local feature information of small targets, but the increase in the downsampling multiple will lead to the loss of global information. Therefore, an additional branch structure is added and a shallower feature map is used to obtain more global feature information.

[0052] Furthermore, the first network model outputs a first classification result of the greening process in the segmented image data, including: obtaining feature information of the target at different scales based on a multi-scale attention module, so that the target is easier to detect and focus on; fusing the feature maps of different network levels based on a lightweight spatial pyramid pooling solution to ensure the balance of features in local and global information; generating the first classification result based on the ratio of the amount of feature pixels in the mask map obtained by the decoding head fusion processing to the area of ​​the entire image.

[0053] Specifically, the EMCA model in the embodiment of the present application performs semantic segmentation on tea images and realizes efficient multi-scale feature extraction by introducing multiple convolutional sub-networks. Secondly, the lightweight spatial pyramid pooling solution extracts feature maps from different downsampling ratios (1 / 2, 1 / 4, 1 / 32) of Backbone for processing, which better takes into account the deep features and shallow features. The deepening processing can obtain more detailed information, while the shallowing processing retains the global feature information.

[0054] It should be noted that the embodiment of the present application can also convert the first network model into ONNX format and deploy it on the Raspberry Pi. The final recognition result of the model can be used to control the Raspberry Pi to send instructions to control the start and stop of the motor and fan, reducing labor costs. The size of the converted model weight file is 6.99M, which is 297% less than before the conversion. Using the ONNXRUNTIME reasoner for reasoning, subject to the hardware performance of the Raspberry Pi itself, the average time for recognizing green pictures with large and obviously concentrated "golden edges" is 8.31s, and the average time for recognizing green pictures with scattered and inconspicuous "golden edges" is 13.56s.

[0055] In one embodiment of the present application, before inputting the image data into the first network model, it also includes: obtaining an image data set of tea leaves, wherein the image data set is divided into a training set, a test set and a verification set; using the training set to perform classification training on the first network model, and using the test set and the verification set to test and verify the trained first network model to verify whether the model structure design achieves the expected effect.

[0056] It is understandable that the embodiment of the present application can adopt a random partitioning method, and all image data in the image data set are divided according to a ratio of 8:1:1, such as 1923 training sets, 240 test sets, and 240 verification sets. In the actual execution process, the embodiment of the present application can adjust the image size to a fixed size of 512×512, and the batch size is 4. The polynomial decay learning rate scheduler is used for adjustment, and the learning rate is gradually adjusted in the form of a polynomial function at different stages of training according to the set parameters. The initial learning rate is 0, the minimum learning rate is 0.0001, and the decay rate is 0.9. The optimizer uses SGD (Stochastic Gradient Descent, random gradient descent), sets the learning rate to 0.01, the momentum to 0.9, and the weight decay to 0.0005; the loss function uses the cross entropy loss function to measure the difference between the predicted value and the true label. The embodiment of the present application can set the maximum number of iterations of training to 160K, save the model every 480 iters, and determine whether the training has reached a stable point through the loss function change curve, and select the best first network model.

[0057] Furthermore, the embodiments of the present application can run a test set on the selected optimal model, calculate indicators such as accuracy and recall, and comprehensively evaluate the model performance. At the same time, some conditions for stopping training can be preset, such as when the loss on the validation set no longer decreases significantly, or when a predetermined maximum number of iterations is reached and there is no improvement for several consecutive rounds, the training can be terminated early to prevent overfitting.

[0058] In order to illustrate the effectiveness of the first network model structure improvement proposed in the application embodiment, an ablation experiment was conducted on the test set, and the results are shown in Table 1.

[0059] Table 1

[0060] Model Parameter quantity MIoU% MPA% MobileNetV3 3.087M 78.02 82.27 +EMCA 1.628M 79.88 84.46 + Improved LRASPP 3.282M 80.05 85.09 +EMCA+Improved LRASPP 1.823M 82.95 90.53

[0061] As can be seen from the table, using the improved LRASPP decoding head to replace the original LRASPP shows higher segmentation accuracy with a slight increase in parameters, indicating that although adding a branch and changing the downsampling multiple will increase the model parameters, it can better integrate feature information of different scales. Using the EMCA attention mechanism to replace the original SE module greatly reduces the model parameters while increasing the segmentation accuracy, indicating that the multi-scale parallel network structure can better capture scattered, inconspicuous, and complex feature information.

[0062] The embodiment of the present application can also conduct a comparative experiment on the proposed first network model and the current mainstream classic semantic segmentation network, and the results are shown in Table 2.

[0063] Table 2

[0064] Model Parameter quantity MIoU% MPA% Swin-Transformer 58.942M 82.90 88.09 UNet 28.987M 81.02 85.11 DeepLablv3+ 41.225M 80.30 83.51 PSPNet 12.639M 79.20 83.28 BiSeNetv2 3.341M 80.15 86.93 Fast-Scnn 1.398M 79.68 84.13 MobileNetV3 3.282M 78.02 82.27 This application method 1.823M 82.95 90.53

[0065] As can be seen from Table 2, compared with the lightweight semantic segmentation algorithms BiSeNetv2, PSPNet, and MobileNet, the parameters of this model are reduced by 44%, 85%, and 45%, but MIoU is increased by 2.8%, 3.75%, and 4.93%, respectively, and MPA is increased by 3.6%, 7.25%, and 8.26%, respectively. Compared with Fast-Scnn, although the parameters have increased, MIoU and MPA are increased by 3.27% and 6.4%, respectively. Compared with the models Swin-Transformer, UNet, and DeepLabv3 with larger parameters, MIoU is only improved by about 2%, but the parameters are only 3%, 6%, and 5% of the model. In summary, this model has the best comprehensive performance, indicating that the use of improved LRASPP can better integrate local information and global information, and EMCA can make the model pay more attention to small and scattered target features in the image, thereby improving the segmentation accuracy of the "golden edge" of tea leaves. The segmentation effects of each network on the test set are as follows: Figure 7 shown.

[0066] In one embodiment of the present application, a second classification result of tea leaves for greening is identified based on temperature and humidity data and weight data, including: inputting the temperature and humidity data and weight data into a second network model, the second network model outputting the weight loss rate of tea leaves during the greening process; and generating a second classification result based on the weight loss rate of tea leaves.

[0067] Among them, the second network model includes: an input layer, a hidden layer and an output layer, wherein a first connection weight is set between the input layer and the hidden layer, a second connection weight is set between the hidden layer and the output layer, and an input gate, an output gate and a forget gate are set inside the hidden layer.

[0068] During the production process, the degree of tea leaf ripening is not judged solely based on the tea leaf image. The tea leaf image, fermentation temperature and humidity, leaf moisture content, leaf aroma (flower and fruit aroma) etc. are all indicators for judging whether the tea leaf is ripe properly. Therefore, the embodiment of the present application combines the temperature, humidity and weight characteristics in the tea leaf ripening process to achieve a more accurate classification of the degree of tea leaf ripening, thereby improving the accuracy of the overall detection.

[0069] Specifically, the second network model can be LSTM (Long Short-Term Memory, long short-term memory network), such as Figure 8 As shown in Figure 1, it consists of three parts: input layer, hidden layer and output layer. t represents the input vector at time t, h t Represents the hidden state vector output at time t. U represents the first connection weight between the input layer and the hidden layer, W represents the connection weight between the hidden layer and the hidden layer, and V represents the second connection weight between the hidden layer and the output layer. In the LSTM structure, input gates, output gates, and forget gates are set inside the hidden layer neurons. These gates determine which information should be remembered through sigmoid functions and bitwise multiplication, giving LSTM a long-term memory function.

[0070] During the actual execution process, the temperature and humidity data and weight data of the embodiment of the present application are input into the long short-term memory network LSTM as input features. The output layer is responsible for calculating the weight loss rate of tea leaves during the greening process and outputting a scalar value to represent the weight loss rate.

[0071] In the embodiment of the present application, temperature, humidity, and weight are feature columns, the label column is the state of green tea leaves, and the leaf weight loss rate is divided into 6 categories, namely, level 1: weight loss rate 0%-4%; level 2: weight loss rate 4%-8%; level 3: weight loss rate 8%-12%; level 4: weight loss rate 12%-16%; level 5: weight loss rate 16%-20%; level 6: weight loss rate 20%-24%.

[0072] In addition, the second network model of the embodiment of the present application also needs to be trained. You can refer to the training method of the above network model, or you can make specific settings according to the above. For example, the weight loss rate of green tea leaves gradually changes with the greening process. Therefore, the six states are subdivided into time periods in the data set label column, and time_step is set to 10, 6000 epochs are trained, the training set and test set are divided into 8:1, the learning rate is set to 0.001, and the batch_size is 256. The prediction results are as follows: Fig. 9 shown.

[0073] In step S103, the degree of greening of the tea leaves is finally determined according to the first classification result and the second classification result.

[0074] The first network of the embodiment of the present application obtains the first classification result through the image features of the tea leaves, and the second network model obtains the second classification result through the temperature, humidity and weight features during the greening process. The classification results obtained by the two network inferences are fused at the decision level to obtain the final classification result. This method can achieve a more accurate greening degree classification based on the semantic information of the greening leaves by combining the temperature, humidity and weight features during the greening process, thereby improving the accuracy of the overall detection.

[0075] It should be noted that due to the different data formats of image data and temperature, humidity and weight data, the decision-level fusion method in multimodal fusion is used for final detection. There are three types of multimodal information fusion strategies, namely pixel-level, feature-level and decision-level fusion. Pixel-level fusion, also known as early fusion, completes pixel fusion by processing multimodal data, but this method requires more detailed processing of multimodal data and the amount of calculation is too large. Feature-level fusion fuses the extracted features through modal interaction or direct fusion, but the difference between the image features and the environmental features is large, which will affect the final prediction results. Decision-level fusion is also known as late fusion, and its structure is as follows: Fig.10 As shown in the figure, this is the highest level of fusion strategy. Decision-level fusion can parallelize information of different modes and classify the final results. It can ensure low computing cost and fast computing speed, and can be deployed on mobile devices to achieve real-time prediction effects.

[0076] Combine the following Fig.11 The multimodal tea leaf greenness detection method according to an embodiment of the present application is described in detail, and mainly consists of two parts: image data processing and temperature, humidity and weight data processing.

[0077] 1) Image data processing

[0078] The image data is collected through a visual device and input into the first network model (such as a tea semantic segmentation model based on MobileNet V3). The tea semantic segmentation model processes the image, separates the tea leaves, and generates a mask map, which is used to calculate the ratio of image features to total image pixels, and then determines the first classification result of the tea leaves.

[0079] 2) The temperature and humidity sensor and the weight sensor collect temperature and humidity data and weight data respectively. The collected data are preprocessed and input into a second network model (such as LSTM). The LSTM model is used to predict the weight loss rate of tea leaves.

[0080] 3) The classification results of the image data are combined with the classification results of the temperature, humidity and weight data to form the final classification results, which can provide a more comprehensive and accurate assessment of the degree of greenness of the tea leaves and improve the efficiency and accuracy of tea quality control.

[0081] According to the method for detecting the degree of greening of tea leaves based on multimodality proposed in the embodiment of the present application, the image data of the tea leaves during the greening process and the temperature, humidity and weight changes are combined, and the first classification result is obtained through the image data of the tea leaves, and the second classification result is obtained through the temperature, humidity and weight characteristics of the greening process. The classification results are fused at the decision level to determine the degree of greening of the tea leaves. Based on the semantic information of the greening leaves, a more accurate classification of the degree of greening can be achieved by combining the temperature, humidity and weight characteristics of the greening process, thereby improving the accuracy of the overall detection. Thus, the problem of detecting tea leaves based on image processing in the related art is solved, the judgment conditions are single, and there are problems such as long time consumption, complex operation and low accuracy.

[0082] Next, a multi-modal tea leaves greenness detection device proposed in accordance with an embodiment of the present application will be described with reference to the accompanying drawings.

[0083] Fig.12 It is a block diagram of a multimodal tea leaves greenness detection device according to an embodiment of the present application.

[0084] like Fig.12 As shown, the multimodal tea greenness detection device 10 includes: an acquisition module 100, an identification module 200 and a determination module 300.

[0085] Among them, the acquisition module 100 is used to obtain image data, temperature and humidity data and weight data of tea leaves during the greening process; the identification module 200 is used to identify the first classification result of the greening leaves according to the image data, and to identify the second classification result of the greening leaves according to the temperature and humidity data and weight data; the determination module 300 is used to determine the degree of greening of the tea leaves according to the first classification result and the second classification result.

[0086] In one embodiment of the present application, the recognition module is further used to input the image data into a first network model, the first network model performs tea semantic segmentation on the image data, and outputs a first classification result of green leaves in the segmented image data, wherein the first network model is improved based on MobileNetV3-Large, and introduces an efficient multi-scale attention module and a lightweight spatial pyramid pool to resolve the problem.

[0087] In one embodiment of the present application, the multi-scale attention module includes: multiple convolutional subnetworks and grouping operations to ensure the ability to extract features in global spatial information, local spatial features, and channel dimensions. The lightweight spatial pyramid pooling solution adopts a three-branch structure to extract feature maps at different sampling ratios, so that the model integrates the advantages of deep and shallow features when fusing features, effectively improving the accuracy of segmentation tasks.

[0088] In one embodiment of the present application, the first network model outputs a first classification result of green leaves in the segmented image data, including: obtaining feature information of the target at different scales based on the multi-scale attention module, so that the target is easier to detect and focus on; fusing the feature maps of different network levels based on the lightweight spatial pyramid pooling solution to ensure the balance of features in local and global information; generating the first classification result according to the ratio of the amount of feature pixels in the mask map obtained by the decoding head fusion processing to the area of ​​the entire image.

[0089] Optionally, a second classification result of tea leaves for greening is identified based on temperature and humidity data and weight data, including: inputting the temperature and humidity data and weight data into a second network model, the second network model outputting the weight loss rate of tea leaves during the greening process; and generating a second classification result based on the weight loss rate of tea leaves.

[0090] In one embodiment of the present application, the second network model includes: an input layer, a hidden layer and an output layer, wherein a first connection weight is set between the input layer and the hidden layer, a second connection weight is set between the hidden layer and the output layer, and an input gate, an output gate and a forget gate are set inside the hidden layer.

[0091] In one embodiment of the present application, the multimodal tea leaf greening degree detection device 10 also includes a training module, which is used to obtain a tea leaf greening image data set before inputting the segmented image data into the first network model, wherein the image data set is divided into a training set, a test set and a verification set; the training set is used to perform classification training on the first network model, and the test set and the verification set are used to test and verify the trained first network model to verify whether the model structure design achieves the expected effect.

[0092] It should be noted that the aforementioned explanation of the embodiment of the method for detecting the greenness of tea leaves based on multi-modality is also applicable to the device for detecting the greenness of tea leaves based on multi-modality in this embodiment, and will not be repeated here.

[0093] According to the multimodal tea-leaf ripening degree detection device proposed in the embodiment of the present application, the image data of the tea-leaf ripening process and the temperature, humidity and weight changes are combined, and the first classification result is obtained through the image data of the tea leaves, and the second classification result is obtained through the temperature, humidity and weight characteristics of the ripening process. The classification results are fused at the decision level to determine the ripening degree of the tea leaves. Based on the semantic information of the ripening leaves, it is possible to achieve a more accurate classification of the ripening degree by combining the temperature, humidity and weight characteristics of the ripening process, thereby improving the accuracy of the overall detection. In this way, the image processing-based method in the related art to detect tea leaves is solved, the judgment conditions are single, and there are problems such as long time consumption, complex operation and low accuracy.

[0094] Fig.13 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

[0095] A memory 1301 , a processor 1302 , and a computer program stored in the memory 1301 and executable on the processor 1302 .

[0096] When the processor 1302 executes the program, the multi-modal tea leaf ripening degree detection method provided in the above embodiment is implemented.

[0097] Furthermore, the electronic device further comprises:

[0098] The communication interface 1303 is used for communication between the memory 1301 and the processor 1302 .

[0099] The memory 1301 is used to store computer programs that can be executed on the processor 1302 .

[0100] The memory 1301 may include a high-speed RAM (Random Access Memory) memory, and may also include a non-volatile memory, such as at least one disk memory.

[0101] If the memory 1301, the processor 1302 and the communication interface 1303 are implemented independently, the communication interface 1303, the memory 1301 and the processor 1302 can be connected to each other through a bus and communicate with each other. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.13 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0102] Optionally, in a specific implementation, if the memory 1301, the processor 1302 and the communication interface 1303 are integrated on a chip, the memory 1301, the processor 1302 and the communication interface 1303 can communicate with each other through an internal interface.

[0103] The processor 1302 may be a CPU (Central Processing Unit), or an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application.

[0104] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned multi-modal tea leaf greenness detection method.

[0105] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms are not necessarily directed to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0106] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0107] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0108] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array, a field programmable gate array, etc.

[0109] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0110] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A method for detecting the degree of greenness of tea leaves based on multimodality, characterized in that: The following steps are involved: Obtain image data, temperature, humidity data and weight data of tea leaves during the greening process; Identify a first classification result of the greening leaves according to the image data, and identify a second classification result of the greening leaves according to the temperature and humidity data and the weight data; The degree of greening of the tea leaves is determined according to the first classification result and the second classification result.

2. The method for detecting the degree of greenness of tea leaves based on multimodality according to claim 1, characterized in that: The first classification result of identifying green leaves according to the image data includes: The image data is input into the first network model, the first network model performs tea semantic segmentation on the image data, and outputs the segmented tea mask image data, and determines the first classification result of green leaves according to the proportion of feature pixels in the mask image, wherein the first network model is based on MobileNetV3-Large, and introduces an efficient multi-scale attention module and a lightweight spatial pyramid pool to resolve the dock.

3. The method for detecting the degree of greenness of tea leaves based on multimodality according to claim 2, characterized in that: The multi-scale attention module includes: multiple convolutional sub-networks and grouping operations, through which features are extracted in global spatial information, local spatial features and channel dimensions; the lightweight spatial pyramid pooling solution adopts a three-branch structure to respectively extract feature maps under different sampling ratios, so that the first network model integrates deep and shallow features at the same time when fusing features.

4. The method for detecting the degree of greenness of tea leaves based on multimodality according to claim 3, characterized in that: The first classification result of the green leaves in the output segmented image data of the first network model includes: Obtaining feature information of the target at different scales based on the multi-scale attention module; Based on the lightweight spatial pyramid pooling solution, the feature graphs of different network levels are fused; The first classification result is generated according to the ratio of the amount of feature pixels in the mask image obtained by the decoding head fusion processing to the area of ​​the entire image.

5. The method for detecting the degree of greenness of tea leaves based on multimodality according to claim 1, characterized in that: The step of identifying the second classification result of the green leaves according to the temperature and humidity data and the weight data comprises: Inputting the temperature and humidity data and the weight data into a second network model, the second network model outputting the weight loss rate of the tea leaves during the greening process; The second classification result is generated according to the blade weight loss rate.

6. The method for detecting the degree of greenness of tea leaves based on multimodality according to claim 5, characterized in that: The second network model includes: an input layer, a hidden layer and an output layer, wherein a first connection weight is set between the input layer and the hidden layer, a second connection weight is set between the hidden layer and the output layer, and an input gate, an output gate and a forget gate are set inside the hidden layer.

7. The method for detecting the degree of greenness of tea leaves based on multimodality according to claim 2, characterized in that: Before inputting the image data into the first network model, the method further includes: Obtaining a tea leaf image dataset, wherein the image dataset is divided into a training set, a test set, and a validation set; The first network model is classified and trained using the training set, and the trained first network model is tested and verified using the test set and the verification set to verify whether the model structure design achieves the expected effect.

8. A multi-modal tea greenness detection device, characterized in that: include: An acquisition module is used to acquire image data, temperature and humidity data, and weight data of tea leaves during the greening process; an identification module, used to identify a first classification result of the greening leaves according to the image data, and to identify a second classification result of the greening leaves according to the temperature and humidity data and the weight data; A determination module is used to determine the degree of greening of the tea leaves according to the first classification result and the second classification result.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the multimodal tea leaf greenness detection method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed, the multimodal tea leaf greenness detection method according to any one of claims 1 to 7 is implemented.

Images