Lettuce heartburn detection method, device and equipment for protected vegetables and medium

Abstract

The invention relates to a facility vegetable-oriented lettuce heartburn detection method, device and equipment and a medium, and the method comprises the steps: enabling each attention head to learn the relation between different image blocks through a plurality of first Transform coding layers, so as to capture the relevance of different regions in a to-be-detected lettuce image, each first Transform coding layer comprises a plurality of multi-head attention coding layers; carrying out matrix multiplication fusion on multiple attention weights from different first Transform coding layers to determine a comprehensive image representation, and screening out a plurality of image block matrixes with maximum discrimination on classification according to the attention head weights of each first Transform coding layer; and inputting the image block matrix with the maximum discrimination into a second Transform coding layer to further extract deep features so as to generate a global feature vector, and inputting the global feature vector into the classifier so as to determine that the lettuce image to be detected is the heartburn lettuce. According to the method, the detection precision of the braised lettuce can be remarkably improved.

Description

Technical Field

[0001] The present application relates to the field of agricultural production, and particularly to a method for detecting lettuce heartburn for protected vegetables, a corresponding device, an electronic device, and a computer-readable storage medium. Background Art

[0002] In a plant factory, since the growth environment of plants is precisely controlled by machinery and equipment, the growth rate of plants is accelerated, and the absorption of calcium by the roots also increases accordingly. The root tip is the main part of the plant for calcium absorption. Free calcium ions in the nutrient solution are easily precipitated with sulfate ions and water-soluble phosphate ions, reducing the availability of calcium. The lack of calcium in newly emerged leaves will cause the cell wall not to form, affecting the meristem of cell tissues, and manifested in symptoms such as the yellowing of young tender heart leaves caused by the damage of the cell membrane and cell wall structure and function, the wrinkling and chlorosis of the leaf margin, and the withering of the leaf tip. This not only greatly reduces the appearance and internal quality of leafy vegetables, but also for the lettuce leaves with heartburn, they need to be processed by manual pruning or removed from the lettuce products, consuming a large amount of labor costs and time. Early methods for detecting heartburn usually rely on manual operation, which is extremely inefficient and difficult to apply to large-scale plant factories.

[0003] In summary, in the prior art, due to the shorter growth cycle and faster growth rate of crops grown in plant factories, it is easy to have calcium deficiency in newly emerged leaves, leading to heartburn. If the manual visual inspection method for heartburn is used, there are problems such as low efficiency, extremely labor-consuming and time-consuming, and it is difficult to detect heartburned lettuce in real time and quickly in the daily production of large-scale plant factories. For this reason, the applicant has made corresponding explorations. Summary of the Invention

[0004] The purpose of the present application is to solve the above problems and provide a method for detecting lettuce heartburn for protected vegetables, a corresponding device, an electronic device, and a computer-readable storage medium.

[0005] To achieve the various purposes of the present application, the following technical solutions are adopted:

[0006] A method for detecting lettuce heartburn for protected vegetables proposed for one of the purposes of the present application includes:

[0007] In response to a lettuce heartburn detection instruction for protected vegetables, obtain a to-be-detected lettuce image containing lettuce heartburn, and input the to-be-detected lettuce image into a trained lettuce heartburn detection model in a converged state, where the lettuce heartburn detection model is constructed by a feature learning network based on a ViT model followed by a classifier;

[0008] In the feature learning network based on the ViT model, the to-be-detected lettuce image is sliced into multiple image patches, and the multiple image patches are mapped into a high-dimensional embedding space through a linear embedding layer, so as to add a position matrix representing the position of the image patch in the to-be-detected lettuce image to each image patch, and initialize a classification matrix;

[0009] After passing through multiple first Transformer encoding layers, each attention head can learn the relationships between different image patches to capture the correlations of different regions in the to-be-detected lettuce image. Among them, each first Transformer encoding layer includes multiple multi-head attention encoding layers;

[0010] The multi-head attention weights from different first Transformer encoding layers are fused through matrix multiplication to determine a comprehensive image representation. According to the multiple attention head weights of each first Transformer encoding layer, multiple image patch matrices with the greatest discriminability for classification are screened out;

[0011] The image patch matrix with the greatest discriminability is input into a second Transformer encoding layer to further extract deep features, so as to generate a global feature vector. The global feature vector is input into the classifier to determine that the to-be-detected lettuce image is a burned lettuce, so as to complete the lettuce heartburn detection for protected vegetables.

[0012] Optionally, the steps of training the feature learning network based on the ViT model include:

[0013] The loss function L(w) is used to calculate and determine the model loss under the current weight w, and backpropagation is performed on it. It is calculated and determined that the derivative of the loss function L(w) with respect to the weight w is And the weight w is updated according to the gradient, which is expressed as:

[0014]

[0015] Among them, L(w) represents the loss function of the current model under the current weight w, w represents the weight of the model, w t represents the weight at the current time step t, w t+1 represents the updated weight at the next time step t + 1, η represents the learning rate, λ represents the weight decay coefficient, represents the gradient of the loss function L(w) with respect to the weight w, which is used to calculate the direction and magnitude of weight update;

[0016] The weight decay term is added to the loss function L(w), and its expression is:

[0017]

[0018] Among them, represents the weight decay term, which prevents overfitting by penalizing large weights;

[0019] Derive the loss function L(w) after adding the weight decay term to determine the weight update derivative, and its expression is:

[0020]

[0021] When updating the weights, use the maximum norm value max_norm to limit the norm of the weights w to ensure that the norm of the weights after each update does not exceed the set maximum threshold, and its expression is:

[0022]

[0023] Among them, max_norm represents the maximum norm value that constrains the weights w.

[0024] Optionally, the steps of training the feature learning network based on the ViT model include:

[0025] Randomly initialize the weights W of the network model 1 , the weight decay coefficient WD 1 and the cross-entropy loss function L CE , obtain the first training round corresponding to the feature learning network based on the ViT model, where the first training round is from t = 1 to t = T 1 ;

[0026] Sample each pair of input x and true label y from a preset data set , perform forward propagation through the network, and calculate the predicted value y of the model hat = f(x);

[0027] Use the cross-entropy loss function L CE to calculate the loss between the predicted value y of the model hat and the true label y;

[0028] Calculate the gradient of the cross-entropy loss function L CE with respect to the weights W 1 through backpropagation and update it, and its expression is:

[0029]

[0030] Repeat the above steps until the number of training times reaches t = T 1 to complete the training of the feature learning network based on the ViT model.

[0031] Optionally, the step of training the classifier includes:

[0032] Initializing the weights W of the classifier 2 , the weight decay coefficient WD 2 , the class balance loss function L CB and the maximum norm value max_norm, obtaining the corresponding second training round of the classifier, where W 2 = W 1 , and the second training round is from t = 1 to t = T 2 ;

[0033] Freezing all model parameters except the classifier, making the classification head weights of the burned lettuce detection model trainable;

[0034] Sampling each pair of input x and true label y from the dataset , and calculating the predicted value y of the model through forward propagation of the network hat = f(x);

[0035] Using the class balance loss function L CB According to the loss between the predicted value y hat and the true label y, different loss weights are assigned to different classes;

[0036] According to the class balance loss function L CB Updating the weights W 2 , which is expressed as:

[0037]

[0038] Using the maximum norm value max_norm to limit the norm of the updated weights W 2 to ensure that the norm of the updated weights W 2 does not exceed the set maximum threshold;

[0039] Repeating the above steps until the number of training times reaches t = T 2 to complete the training of the classifier.

[0040] Optionally, in the class balance loss function L CB introduce a weight factor α, which is used to weight the loss according to the sample size of the class, and the weight factor α is expressed as:

[0041]

[0042] Among them, β represents a hyperparameter that controls the distribution of category samples, and its value is less than 1. n represents the number of samples in the current category, and N represents the total number of categories. The weight factor α is used to give more attention and weighting to a category when the sample size of that category is small.

[0043] Optionally, in the category balance loss function L CB A classification difficulty factor m is introduced, and the classification difficulty factor m is expressed as:

[0044] m = exp(-γ[y×y hat +log…(1+exp(-y hat ))]);

[0045] Among them, γ is a hyperparameter used to adjust the sensitivity of the model to difficult-to-classify samples, and y represents the true label.

[0046] Optionally, the number of the first Transformer encoding layers is 12, and each of the first Transformer encoding layers contains 12 multi-head attention encoding layers.

[0047] A lettuce heartburn detection device for protected vegetables provided to meet another object of the present application includes:

[0048] A detection model construction module, configured to respond to a lettuce heartburn detection instruction for protected vegetables, obtain a to-be-detected lettuce image containing lettuce heartburn, and input the to-be-detected lettuce image into a trained lettuce heartburn detection model that has converged, where the lettuce heartburn detection model is constructed by a feature learning network based on a ViT model followed by a classifier;

[0049] A position matrix determination module, configured to slice the to-be-detected lettuce image into multiple image patches in the feature learning network based on the ViT model, map the multiple image patches to a high-dimensional embedding space through a linear embedding layer, add a position matrix representing the position of each image patch in the to-be-detected lettuce image to each image patch, and initialize a classification matrix;

[0050] An image patch association module, configured to pass through multiple first Transformer encoding layers, so that each attention head can learn the relationship between different image patches to capture the relevance of different regions in the to-be-detected lettuce image, where each first Transformer encoding layer contains multiple multi-head attention encoding layers;

[0051] The comprehensive image representation determination module is configured to fuse the multi-head attention weights from different first Transformer encoding layers through matrix multiplication to determine a comprehensive image representation, and screen out multiple image patch matrices with the greatest discriminability for classification according to the multiple attention head weights of each first Transformer encoding layer;

[0052] The burned lettuce detection module is configured to input the image patch matrix with the greatest discriminability into a second Transformer encoding layer to further extract deep features, so as to generate a global feature vector, and input the global feature vector into the classifier to determine that the lettuce image to be detected is burned lettuce, so as to complete the detection of burned lettuce in protected vegetables.

[0053] An electronic device provided to meet another object of the present application includes a central processing unit and a memory. The central processing unit is used to call and run a computer program stored in the memory to execute the steps of the method for detecting burned lettuce in protected vegetables according to the present application.

[0054] A computer-readable storage medium provided to meet another object of the present application stores a computer program implemented according to the method for detecting burned lettuce in protected vegetables in the form of computer-readable instructions. When the computer program is called and run by a computer, it executes the steps included in the corresponding method.

[0055] Compared with the prior art, in view of the problems in the prior art that crops grown in plant factories have a shorter growth cycle, a faster growth rate, and are prone to calcium deficiency in new leaves, resulting in heartburn. If the manual visual inspection method is used to detect heartburn, there are problems such as low efficiency, extremely laborious and time-consuming, and it is difficult to detect burned lettuce in real time and quickly in the daily production of large plant factories. The present application includes but is not limited to the following beneficial effects:

[0056] First, the ViT (Vision Transformer) model can better capture the global features of images by using the self-attention mechanism, especially suitable for the processing of large-scale image datasets. Compared with CNN, ViT does not rely on local convolution kernels and can effectively capture features in a larger range, which helps to identify the relatively subtle symptom of lettuce heartburn;

[0057] Second, the overlapping segmentation can solve the local features of small regions in the image, making the model more accurate in balancing local and global information. The partial selection module can screen specific regions of the image according to importance, improving the model's ability to identify heartburn symptoms;

[0058] Thirdly, through contrastive learning, the model can enhance its sensitivity to heartburn symptoms by comparing the similarities between different images. Meanwhile, the weight balancing technique helps to address the issue of the imbalance between the number of heartburn samples and healthy samples, avoiding the model's bias towards identifying healthy lettuce and improving the accuracy of heartburn detection.

[0059] Furthermore, the lettuce heartburn detection method for protected vegetables in this application can effectively reduce the manual workload, provide more reliable detection results, and through hardware support and a real-time feedback system, make the entire detection process more efficient and operable. This technology can not only improve the management efficiency of plant factories but also provide strong support for the automated development of modern agriculture. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The above and / or additional aspects and advantages of this application will become apparent and be readily understood from the following description of the embodiments in conjunction with the drawings, where:

[0061] Figure 1 is a flowchart of the lettuce heartburn detection method for protected vegetables in an embodiment of this application;

[0062] Figure 2 is a schematic diagram of the lettuce heartburn detection device for protected vegetables in an embodiment of this application;

[0063] Figure 3 is an exemplary network architecture of the heartburn lettuce detection model in an embodiment of this application;

[0064] Figure 4 is a principle block diagram of the lettuce heartburn detection device for protected vegetables in an embodiment of this application;

[0065] Figure 5 is a schematic diagram of the structure of the computer device in an embodiment of this application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0066] The embodiments of this application will be described in detail below. The examples of the embodiments are shown in the drawings, where the same or similar reference numerals denote the same or similar elements or elements with the same or similar functions throughout. The embodiments described below by referring to the drawings are exemplary and are only used to explain this application and should not be construed as limiting this application.

[0067] Those skilled in the art can understand that, unless specifically stated otherwise, the singular forms "a", "an", "the" and "said" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the specification of this application means the presence of the stated features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or their groups. It should be understood that when we say that an element is "connected" or "coupled" to another element, it can be directly connected or coupled to other elements, or there may also be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The phrase "and / or" used herein includes all or any unit and all combinations of one or more of the associated listed items.

[0068] Those skilled in the art can understand that, unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as the general understanding of those of ordinary skill in the art to which this application belongs. It should also be understood that terms such as those defined in a general dictionary should be understood to have a meaning consistent with the meaning in the context of the prior art, and will not be interpreted in an idealized or overly formal sense unless specifically defined as here.

[0069] Those skilled in the art can understand that the "client", "terminal", and "terminal device" used herein include both devices with a wireless signal receiver that only has the ability to receive and no ability to transmit, and devices with receiving and transmitting hardware that have the receiving and transmitting hardware capable of two-way communication on a two-way communication link. Such devices may include: cellular or other communication devices such as personal computers, tablet computers, etc., which have a single-line display or a multi-line display or a cellular or other communication device without a multi-line display; PCS (Personal Communications Service), which can combine voice, data processing, fax, and / or data communication capabilities; PDA (Personal Digital Assistant), which may include a radio frequency receiver, pager, Internet / intranet access, web browser, notepad, calendar, and / or GPS (Global Positioning System) receiver; conventional laptop and / or palm-held computers or other devices, which are conventional laptop and / or palm-held computers or other devices with and / or including a radio frequency receiver. The "client", "terminal", and "terminal device" used herein can be portable, transportable, installed in a vehicle (air, sea, and / or land), or suitable for and / or configured to run locally, and / or run in a distributed form at any other location on the earth and / or in space. The "client", "terminal", and "terminal device" used herein can also be a communication terminal, an Internet access terminal, a music / video playback terminal, for example, it can be a PDA, MID (Mobile Internet Device), and / or a mobile phone with music / video playback function, or it can also be a smart TV, a set-top box, etc.

[0070] The hardware referred to by names such as "server", "client", and "service node" in this application is essentially an electronic device with the equivalent capabilities of a personal computer, and is a hardware device with the necessary components revealed by the von Neumann principle, including a central processing unit (including an arithmetic unit and a controller), a memory, an input device, and an output device. The computer program is stored in its memory, and the central processing unit loads the program stored in the external memory into the memory for execution, executes the instructions in the program, and interacts with the input / output devices to complete specific functions.

[0071] It should be noted that the concept of "server" referred to in this application can similarly be extended to the case applicable to a server cluster. According to the network deployment principle understood by those skilled in the art, the servers should be logically divided. Physically, these servers can either be independent of each other but can be called through an interface, or integrated into a physical computer or a set of computer clusters. Those skilled in the art should understand this flexibility and should not be restricted by this when implementing the network deployment method of this application.

[0072] One or several technical features of this application, unless expressly specified, can either be deployed on the server for the client to remotely call and obtain the online service interface provided by the server to implement access, or directly be deployed and run on the client to implement access.

[0073] The neural network models cited or possibly cited in this application, unless expressly specified, can either be deployed on a remote server and remotely called on the client, or be deployed on a client capable of handling the device for direct calling. In some embodiments, when it runs on the client, its corresponding intelligence can be obtained through transfer learning to reduce the requirements for the client's hardware operation resources and avoid over-occupying the client's hardware operation resources.

[0074] All kinds of data involved in this application, unless expressly specified, can either be remotely stored on the server or stored on the local terminal device, as long as it is suitable for being called by the technical solution of this application.

[0075] Those skilled in the art should be aware of this: Although the various methods of this application are described based on the same concept and thus show commonality with each other, unless otherwise specified, these methods can all be executed independently. Similarly, for each embodiment disclosed in this application, they are all proposed based on the same inventive concept. Therefore, for the concepts with the same expression, as well as the concepts that are only appropriately transformed for convenience although the concept expressions are different, they should be equivalently understood.

[0076] For each embodiment to be disclosed in this application, unless expressly pointed out that there is a mutually exclusive relationship between them, otherwise, the relevant technical features involved in each embodiment can be cross-combined to flexibly construct new embodiments, as long as this combination does not deviate from the creative spirit of this application and can meet the needs in the prior art or solve certain deficiencies in the prior art. Those skilled in the art should be aware of this flexibility.

[0077] Please refer to Figure 1 , in one embodiment of the method for detecting lettuce heartburn for protected vegetables of this application, this method is applicable to the detection of lettuce heartburn in protected agriculture and has been preliminarily tested in a plant factory, including:

[0078] Step S10: In response to the instruction for detecting lettuce heartburn in protected vegetables, obtain the lettuce image to be detected that contains lettuce heartburn, and input the lettuce image to be detected into the trained lettuce heartburn detection model that has converged. Among them, the lettuce heartburn detection model is constructed by connecting a feature learning network based on the ViT model to a classifier;

[0079] Step S20: In the feature learning network based on the ViT model, divide the lettuce image to be detected into multiple image patches, map the multiple image patches to a high-dimensional embedding space through a linear embedding layer, add a position matrix representing the position of each image patch in the lettuce image to be detected, and initialize a classification matrix;

[0080] Step S30: After passing through multiple first Transformer encoding layers, enable each attention head to learn the relationships between different image patches to capture the correlations between different regions in the lettuce image to be detected. Among them, each first Transformer encoding layer includes multiple multi-head attention encoding layers;

[0081] Step S40: Perform matrix multiplication fusion on the multi-head attention weights from different first Transformer encoding layers to determine a comprehensive image representation. According to the weights of multiple attention heads in each first Transformer encoding layer, screen out multiple image patch matrices that have the greatest discriminability for classification;

[0082] Step S50: Input the image patch matrix with the greatest discriminability into a second Transformer encoding layer to further extract deep features, generate a global feature vector, and input the global feature vector into the classifier to determine that the lettuce image to be detected is lettuce with heartburn, so as to complete the detection of lettuce heartburn in protected vegetables.

[0083] Please refer to Figure 2 , the method for detecting lettuce heartburn in protected vegetables of the present application can be implemented based on a lettuce heartburn detection device for protected vegetables that can move within the multi-layer planting rack track of a plant factory. Among them, the lettuce heartburn detection device for protected vegetables includes two industrial cameras and an image processing module. The industrial cameras obtain the growth images of lettuce in the plant factory through the way of taking overhead pictures. The image processing module can deploy the lettuce heartburn detection model to analyze the overhead pictures obtained by the industrial cameras in real time and perform the heartburn detection of lettuce to detect lettuce with heartburn;

[0084] Specifically, the image processing module in the lettuce heartburn detection device for protected vegetables can respond to the lettuce heartburn detection instruction for protected vegetables, obtain the to-be-detected lettuce image containing lettuce heartburn, and input the to-be-detected lettuce image into the trained-to-converge heartburn lettuce detection model, where the heartburn lettuce detection model is constructed by connecting a feature learning network based on the ViT model to a classifier;

[0085] In some embodiments, in a plant factory, according to the spatial dimensions of the multi-layer planting rack, it is designed by imitating the existing planting tray moving device in the plant factory. The STM32 control unit and the handheld remote control are used to control the forward and reverse rotation of the motor to realize the forward and backward movement of the device. Guide wheels are installed at the front and rear of the device to fit the track to ensure that the device does not deviate from the track during driving.

[0086] In a further embodiment, the steps of using the above industrial camera to obtain the to-be-detected lettuce image include: there are 32 lettuce plants in 4 rows and 8 columns in each planting tray. According to the planting positions of the lettuce in the planting tray, the left and right parts of the planting tray are respectively photographed by two cameras on the left and right. The installation positions of the cameras are adjusted so that each camera obtains an image containing 16 lettuce plants in 4 rows and 4 columns; Taking loose-leaf lettuce as the research object to carry out heartburn detection research. Through preliminary observation, about 20 days before and after the lettuce is transplanted into the planting tray is the high-incidence period of heartburn. Image acquisition starts when slight heartburn is observed manually.

[0087] In a further embodiment, after collecting the images, the to-be-detected lettuce image containing 16 plants in 4×4 is manually cut into 16 images containing only one lettuce plant, and the heartburn lettuce and healthy lettuce are classified by careful visual observation. All the lettuce images are divided into two categories: heartburn and healthy. Since the background of the lettuce images in the plant factory is a white plastic board, which is relatively simple, the target detection network model can be used to replace the manual cutting in the follow-up to achieve the effect of full-process automatic detection; The classified images are made into a dataset in the CUB-200-2011 format, and the training set and the validation set are divided in a ratio of 8:2.

[0088] Specifically, in the early stage, a lettuce heartburn detection device for protected vegetables was designed according to the spatial dimensions of the multi-layer planting rack and the device was built. The growth images of loose-leaf lettuce were collected in a certain plant factory. Among them, there are X planting racks in the plant factory, each planting rack has Y layers, each layer has Z planting trays, and each planting tray has 32 plants in 4 rows and 8 columns. There are tracks between each layer of the planting rack for moving the planting trays. The monitoring device of the present invention also moves on this track and photographs the planting trays below. Approximately 20 days after the lettuce seedlings are transplanted into the planting trays, when slight heartburn is observed with the naked eye, image data collection starts and stops shooting until obvious heartburn appears. The positions of the heartburn lettuce are recorded while shooting images for subsequent production of the classification dataset.

[0089] The acquired images are segmented into single-lettuce images, and a classification dataset is created; according to the recorded heartburn positions during shooting, 1210 heartburn lettuce images and 3939 healthy lettuce images are obtained by manually segmenting and screening 430 pictures containing multiple lettuces. Manual segmentation can be replaced by an object detection model in the future to achieve unmanned heartburn detection throughout the process. A classification dataset in the format of the CUB-200-2011 dataset is created from 1210 heartburn lettuces and 3939 healthy lettuces, and is divided into a training set and a validation set in an 8:2 ratio.

[0090] In a further embodiment, please refer to Figure 3 , the dataset is input into the constructed heartburn lettuce detection model for the first-stage training. In the first stage, the feature learning network based on the ViT model mainly completes the learning of image features; the input is a single-lettuce image with a resolution of 448×448. Using partially overlapping segmentation, while protecting the integrity of adjacent features, the input image is segmented into multiple small image patches. Then, a linear embedding layer is used to map the image patches to a high-dimensional embedding space. A learnable position matrix representing the position of the image patch in the image is added to each image patch matrix, and a classification matrix for subsequent classification is randomly initialized.

[0091] The number of the first Transformer encoding layers is 12. Among them, each first Transformer encoding layer contains 12 multi-head attention encoding layers. After 12 transformer encoding layers, each first Transformer encoding layer contains 12 multi-head attention encoding layers to learn the image features of different image patches and the relationships between all image patches. The image patch matrix after passing through the encoding layer is input into a partial selection module. This module performs matrix multiplication on the multi-head attention weights of different encoding layers, fuses the information of different encoding layers, and selects 12 image patch matrices with the greatest discriminability for classification according to the weights of 12 attention heads in each encoding layer, eliminating a large number of similar but unhelpful image regions for classification.

[0092] The 12 selected image patch matrices and the classification matrix are passed through another transformer encoding layer, that is, the second transformer encoding layer, so that the classification matrix focuses on the selected image regions and the subtle differences between different classes.

[0093] Two-part loss processing is performed on the classification matrix. First, the classification matrix is projected to the output dimensions of two classes through a linear layer to output the probabilities of the two classes respectively, and the cross-entropy loss function is used to supervise the learning of classification. The other part is to use contrastive feature learning on the classification matrix itself to minimize the similarity of the classification matrices between different classes and maximize the similarity of the classification matrices between the same classes.

[0094] In the above training process of the heartburn lettuce detection model, weight decay is added to encourage the model to learn small weights, which enhances the generalization of the model and reduces the imbalance of classification tendency caused by the large difference in the number of category samples.

[0095] In a further embodiment, the model weights trained by the feature learning network based on the ViT model (Vision Transformer) in the first stage are used to train the classifier in the second stage. The overall model architecture in the second stage is similar to that in the first stage, and the purpose is to complete the learning of the balanced classifier.

[0096] In the training of the second stage, all layers except the classifier are frozen, and no gradient updates are performed on these layers. The class-balanced loss function is used for the classifier (the class-balanced loss function is not used in the first stage because it artificially enlarges the calculation gradient of the samples with small quantities, affecting feature learning), and a larger weight decay coefficient than that in the first stage and a maximum weight norm limit are added. Weight screening makes the weights obtained by the model all smaller, and the maximum norm limit restricts the maximum value of the weight change. The combination of the two can solve the over-classification of the categories with large sample quantities and the under-classification of the categories with small sample quantities.

[0097] Please refer to Figure 3 , the strategy of using two-stage decoupled feature learning and classification head learning to train the model in this application has a good effect on the classification task with unbalanced sample quantities after two-stage training; among them, in the first stage, smaller weight decay and standard cross-entropy loss are used for feature learning, and in the second stage, slightly larger weight decay, maximum weight norm limit, and class-balanced loss function are used for classifier learning.

[0098] The above dataset is input into the constructed network model for the first-stage training. The input is a single-plant lettuce image with a resolution of 600×600. Using partially overlapping segmentation, while protecting the integrity of adjacent features, the input image is cut into multiple small image patches. Then, a linear embedding layer is used to map the image patches to a high-dimensional embedding space, a learnable position matrix representing the position of the image patch in the image is added to each image patch matrix, and a classification matrix for subsequent classification is randomly initialized.

[0099] After 12 Transformer encoding layers, each encoding layer including 12 attention heads, learn the image features of different image patches and the relationships among all image patches. Input the image patch matrix after passing through the encoding layers into the partial selection module. This module performs matrix multiplication on the multi-head attention weights of different encoding layers, fuses the information of different encoding layers, and selects 12 image patch matrices with the greatest discriminability for classification according to the weights of the 12 attention heads in each encoding layer, eliminating a large number of similar but unhelpful image regions for classification.

[0100] Pass the 12 selected image patch matrices and the classification matrix through another Transformer encoding layer, enabling the classification matrix to focus on the selected image regions and the subtle differences between different classes. Perform two-part loss processing on the classification matrix. First, project the classification matrix to the output dimensions of two classes through a linear layer to output the probabilities of the two classes respectively, and use the cross-entropy loss function to supervise the learning of classification. The other part is to use contrastive feature learning on the classification matrix itself. By defining images of the same class as positive samples and images of different classes as negative samples, minimize the similarity of the classification matrices of negative samples and maximize the similarity of the classification matrices among positive samples. Add weight decay to the entire above-mentioned model training process to encourage the model to learn small weights, enhancing the generalization of the model while reducing the classification tendency imbalance caused by the large difference in the number of class samples.

[0101] In some embodiments, the steps of training the feature learning network based on the ViT model include:

[0102] Use the loss function L(w) to calculate and determine the model loss under the current weight w, and perform backpropagation on it to calculate and determine the derivative of the loss function L(w) with respect to the weight w as And update the weight w according to the gradient, which is expressed as:

[0103]

[0104] where L(w) represents the loss function of the current model under the current weight w, w represents the weight of the model, w t represents the weight at the current time step t, w t+1 represents the updated weight at the next time step t + 1, η represents the learning rate, λ represents the weight decay coefficient, represents the gradient of the loss function L(w) with respect to the weight w, which is used to calculate the direction and magnitude of weight update;

[0105] Specifically, use the loss function L(w) to calculate the model loss under the current weight, calculate the derivative of L(w) with respect to w as Then update w to determine Then use w t+1Conduct a new round of training.

[0106] Add the weight decay term to the loss function L(w), and its expression is:

[0107]

[0108] where represents the weight decay term, which prevents overfitting by penalizing large weights;

[0109] Specifically, when calculating the derivative of the loss function with respect to the weights, a "penalty" term, i.e., in the formula, is added, which makes the new weights decrease by ηλ×w t , where η is the learning rate and λ is the weight decay coefficient, both of which are hyperparameters. The larger λ is, the smaller the updated weights are, and the smaller the overall weights obtained from the final training are. Adding the "penalty" not only changes the magnitude of the updated weights but also changes the direction of weight updates. The updates aim to minimize the loss. The added "penalty" is a circle centered at the origin in the loss function, which pulls the weights towards the origin, i.e., makes the weights approach zero and deviates from the update direction of the original loss.

[0110] Take the derivative of the loss function L(w) after adding the weight decay term to determine the weight update derivative, and its expression is:

[0111]

[0112] When updating the weights, use the maximum norm value max_norm to limit the norm of the weights w to ensure that the norm of the weights after each update does not exceed the set maximum threshold, and its expression is:

[0113]

[0114] where max_norm represents the maximum norm value that constrains the weights w.

[0115] Specifically, the maximum norm value max_norm limits the maximum value of |w| 2 . When the weight norm |w| 2 is greater than the set preset threshold, set the weight norm |w| 2 to the size of the preset threshold, or recalculate the weight value through the calculation formula . MaxNorm does not change the direction of weight updates but changes the magnitude of weight updates. It is not only helpful for solving the classification problem of unbalanced sample sizes but also improves the generalization ability of the model.

[0116] In a further embodiment, the burned lettuce detection model of the present application adopts a two-stage training strategy. In the first stage, the feature learning network based on the ViT model mainly completes the learning of image features. In the second stage, the classifier is learned. Different weight balancing parameters and loss functions are used in the two stages, but the same instance balancing sampling strategy is used. Using the simplest instance balancing sampling can achieve the optimal effect on the unbalanced dataset through the decoupling method.

[0117] In a specific embodiment, the steps of training the feature learning network based on the ViT model include:

[0118] Step S101: Randomly initialize the weight W of the network model 1 , the weight decay coefficient WD 1 and the cross-entropy loss function L CE , and obtain the first training round corresponding to the feature learning network based on the ViT model, where the first training round is from t = 1 to t = T 1 ;

[0119] Step S102: Sample each pair of input x and true label y from the preset dataset , perform forward propagation through the network, and calculate the predicted value y of the model hat = f(x);

[0120] Step S103: Use the cross-entropy loss function L CE to calculate the loss between the predicted value y of the model hat and the true label y;

[0121] Step S104: Calculate the gradient of the cross-entropy loss function L CE with respect to the weight W 1 through backpropagation and update it, which is expressed as:

[0122]

[0123] Step S105: Repeat the above steps until the number of training times reaches t = T 1 to complete the training of the feature learning network based on the ViT model.

[0124] In a further embodiment, the steps of training the classifier include:

[0125] Step S1001: Initialize the weight W of the classifier 2 , the weight decay coefficient WD 2 , the class balance loss function L CBand the maximum norm value max_norm, obtain the second training round corresponding to the classifier, where W 2 = W 1 , and the second training round is from t = 1 to t = T 2 ;

[0126] Step S1002: Freeze all model parameters except the classifier, and make the classification head weights of the burned lettuce detection model trainable;

[0127] Step S1003: Sample each pair of input x and true label y from the dataset , and through the forward propagation of the network, calculate the predicted value y hat = f(x);

[0128] Step S1004: Adopt the class-balanced loss function L CB According to the loss between the predicted value y hat and the true label y, assign different loss weights to different classes;

[0129] Step S1005: Update the weight W CB according to the class-balanced loss function L 2 , which is expressed as:

[0130]

[0131] Step S1006: Use the maximum norm value max_norm to limit the norm of the updated weight W 2 to ensure that the norm of the weight W 2 after each update does not exceed the set maximum threshold;

[0132] Step S1007: Repeat the above steps until the number of training times reaches t = T 2 to complete the training of the classifier.

[0133] Based on any of the above embodiments, introduce a weight factor α in the class-balanced loss function L CB , which is used to weight the loss according to the sample size of the class. The weight factor α is expressed as:

[0134]

[0135] where β represents a hyperparameter that controls the class sample distribution, and its value is less than 1. n represents the sample number of the current class, and N represents the total number of classes. The weight factor α is used to give more attention and weighting to a class when the sample size of that class is small.

[0136] Specifically, the larger the class sample size n, The smaller it is, the smaller the obtained α after normalization, and the smaller the class sample size n. The larger it is, the larger α is, enabling classes with smaller sample sizes to receive more attention.

[0137] Based on any of the above embodiments, in the class balance loss function L CB introduce a classification difficulty factor m, which is expressed as:

[0138] m = exp(-γ[y × y hat + log(1 + exp(-y hat ))]);

[0139] where γ is a hyperparameter used to adjust the sensitivity of the model to difficult-to-classify samples, and y represents the true label.

[0140] Specifically, m aims to increase the attention to difficult-to-classify samples. The larger y hat and y are, the more difficult the classification is. y is the true label value and has been one-hot encoded, with only positive and negative classes.

[0141] When the class is the positive class, y = 1, and its

[0142] m = exp(-γ[log(1 + exp(-y hat ))]) = exp(-γ[log(1 + exp(-y hat ))]),

[0143] y hat The closer it is to 0, that is, the greater the classification difficulty, the smaller log(1 + exp(y hat )) is, the larger m is, and the loss is amplified. The closer y hat is to 1, the smaller m is, and the impact on the loss is smaller.

[0144] When the class is the negative class, y = 0,

[0145] m = exp(-γ[log(1 + exp(-y hat ))]) = exp(-γ[log(1 + exp(-y hat ))]), y hat The closer it is to 1, that is, the greater the classification difficulty, the smaller log(1 + exp(-y hat )) is, the larger m is, and the loss is amplified. The closer y hat is to 0, the smaller m is, and the impact on the loss is smaller.

[0146] As can be seen from the above embodiments, after the burnt heart lettuce detection model is trained to the convergence state, it can be put into production use. By inputting the lettuce image to be detected into the burnt heart lettuce detection model that has been trained to the convergence state, the burnt heart lettuce in the plant factory can be detected.

[0147] As can be seen from the above embodiments, compared with the prior art, in the prior art, due to the shorter growth cycle and faster growth rate of the crops planted in the plant factory, it is easy to have calcium deficiency in the newly grown leaves, resulting in burnt heart. If the method of detecting burnt heart by the naked eye is used, there are problems such as low efficiency, extremely labor-consuming and time-consuming, and it is difficult to detect the burnt heart lettuce in the daily production of large-scale plant factories in real time and quickly. The present application includes but is not limited to the following beneficial effects:

[0148] First, the ViT (Vision Transformer) model can better capture the global features of the image by using the self-attention mechanism, especially suitable for the processing of large-scale image datasets. Compared with CNN, ViT does not rely on local convolution kernels and can effectively capture features in a larger range, which helps to identify the relatively subtle symptom of lettuce burnt heart.

[0149] Second, the overlapping segmentation can solve the local features of small regions in the image, making the model more accurate in balancing local and global information. The partial selection module can screen specific regions of the image according to importance, improving the model's ability to identify the burnt heart symptom.

[0150] Third, through contrastive learning, the model can enhance the sensitivity to the burnt heart symptom by comparing the similarities between different images. At the same time, the weight balancing technology helps to deal with the problem of imbalance between the burnt heart sample size and the healthy sample size, avoiding the model's bias towards the recognition of healthy lettuce and improving the accuracy of burnt heart detection.

[0151] Furthermore, the lettuce burnt heart detection method for protected vegetables of the present application can effectively reduce the manual workload, provide more reliable detection results, and through hardware support and a real-time feedback system, make the whole detection process more efficient and operable. This technology can not only improve the management efficiency of the plant factory, but also provide strong support for the automated development of modern agriculture.

[0152] Please refer to Figure 4, a lettuce heartburn detection device for facility vegetables provided to meet one of the purposes of this application, includes a detection model construction module 1100, a position matrix determination module 1200, an image patch association module 1300, a comprehensive image representation determination module 1400, and a heartburn lettuce detection module 1500. Among them, the detection model construction module 1100 is configured to respond to the lettuce heartburn detection instruction for facility vegetables, obtain the lettuce image to be detected containing lettuce heartburn, and input the lettuce image to be detected into the trained heartburn lettuce detection model in a converged state. Among them, the heartburn lettuce detection model is constructed by a feature learning network based on the ViT model followed by a classifier; the position matrix determination module 1200 is configured to divide the lettuce image to be detected into multiple image patches in the feature learning network based on the ViT model, map the multiple image patches to a high-dimensional embedding space through a linear embedding layer, so as to add a position matrix representing the position of the image patch in the lettuce image to be detected to each image patch, and initialize a classification matrix; the image patch association module 1300 is configured to pass through multiple first Transformer encoding layers, so that each attention head can learn the relationship between different image patches to capture the relevance of different regions in the lettuce image to be detected. Among them, each first Transformer encoding layer includes multiple multi-head attention encoding layers; the comprehensive image representation determination module 1400 is configured to perform matrix multiplication fusion on the multi-head attention weights from different first Transformer encoding layers to determine a comprehensive image representation, and screen out multiple image patch matrices with the greatest discriminability for classification according to the weights of multiple attention heads of each first Transformer encoding layer; the heartburn lettuce detection module 1500 is configured to input the image patch matrix with the greatest discriminability into a second Transformer encoding layer to further extract deep features to generate a global feature vector, and input the global feature vector into the classifier to determine that the lettuce image to be detected is heartburn lettuce, so as to complete the lettuce heartburn detection for facility vegetables.

[0153] Based on any embodiment of this application, please refer to Figure 5 , another embodiment of this application further provides an electronic device, which can be implemented by a computer device, such as Figure 5As shown, it is a schematic diagram of the internal structure of a computer device. The computer device includes a processor, a computer-readable storage medium, a memory, and a network interface connected through a system bus. Among them, the computer-readable storage medium of the computer device stores an operating system, a database, and computer-readable instructions. The database can store a control information sequence. When the computer-readable instructions are executed by the processor, the processor can implement a method for detecting lettuce heartburn in protected vegetables. The processor of the computer device is used to provide computing and control capabilities to support the operation of the entire computer device. The memory of the computer device can store computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor can execute the method for detecting lettuce heartburn in protected vegetables of the present application. The network interface of the computer device is used to connect and communicate with a terminal. Those skilled in the art can understand that Figure 5 the structure shown in is only a block diagram of some structures related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0154] In this embodiment, the processor is used to execute Figure 4 the specific functions of each module in. The memory stores the program code and various types of data required to execute the above modules. The network interface is used for data transmission between the user terminal or the server. The memory in this embodiment stores the program code and data required to execute all modules / sub-modules in the device for detecting lettuce heartburn in protected vegetables of the present application. The server can call the program code and data of the server to execute the functions of all modules.

[0155] The present application also provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors are caused to execute the steps of the method for detecting lettuce heartburn in protected vegetables according to any embodiment of the present application.

[0156] The present application also provides a computer program product, including a computer program / instructions. When the computer program / instructions are executed by one or more processors, the steps of the method for detecting lettuce heartburn in protected vegetables according to any embodiment of the present application are implemented.

[0157] Those of ordinary skill in the art can understand that all or part of the processes in the above-described embodiments of the method of the present application can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-described methods. Among them, the aforementioned storage medium can be a computer-readable storage medium such as a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM), etc.

[0158] The above are only some embodiments of the present application. It should be noted that for those of ordinary skill in the art of the present technology, without departing from the principle of the present application, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present application.

Claims

1. A method for detecting heartburn in lettuce for greenhouse vegetables, characterized in that: include: In response to a lettuce heartburn detection instruction for facility vegetables, an image of a lettuce to be detected containing a heartburn of the lettuce is obtained, and the image of the lettuce to be detected is input into a heartburn lettuce detection model that has been trained to a convergence state, wherein the heartburn lettuce detection model is constructed by a feature learning network based on a ViT model followed by a classifier; In the feature learning network based on the ViT model, the lettuce image to be detected is divided into multiple image blocks, and the multiple image blocks are mapped to a high-dimensional embedding space through a linear embedding layer, so that a position matrix representing the position of the image block in the lettuce image to be detected is added to each image block, and a classification matrix is ​​initialized; After multiple first Transformer encoding layers, each attention head can learn the relationship between different image blocks to capture the correlation between different regions in the lettuce image to be detected, wherein each first Transformer encoding layer includes multiple multi-head attention encoding layers; The multi-head attention weights from different first Transformer coding layers are fused by matrix multiplication to determine a comprehensive image representation. According to the multiple attention head weights of each first Transformer coding layer, multiple image block matrices with the greatest discriminability for classification are screened out; The image block matrix with the maximum discriminability is input into a second Transformer encoding layer to further extract deep features to generate a global feature vector, and the global feature vector is input into the classifier to determine that the lettuce image to be detected is heartburn lettuce, so as to complete the lettuce heartburn detection for facility vegetables.

2. The method for detecting heartburn in lettuce for greenhouse vegetables according to claim 1, characterized in that: The steps to train a feature learning network based on the ViT model include: The loss function L(w) is used to calculate the model loss under the current weight w, and back propagation is performed on it to calculate the derivative of the loss function L(w) with respect to the weight w: And update the weight w according to the gradient, which is expressed as: Among them, L(w) represents the loss function of the current model under the current weight w, w represents the weight of the model, and w t represents the weight of the current time step t, w t+1 represents the updated weight at the next time step t+1, η represents the learning rate, λ represents the weight decay coefficient, Represents the gradient of the loss function L(w) with respect to the weight w, which is used to calculate the direction and magnitude of the weight update; The weight decay term Added to the loss function L(w), its expression is expressed as: in, represents the weight decay term, which prevents overfitting by penalizing large weights; Add weight decay term The loss function L(w) is derived to determine the weight update derivative, which is expressed as: When updating the weight, the maximum norm value max_norm is used to limit the norm of the weight w to ensure that the norm of the weight after each update does not exceed the set maximum threshold. The expression is expressed as: Among them, max_norm represents the maximum norm value of the constraint weight w.

3. The method for detecting heartburn in lettuce for greenhouse vegetables according to claim 1, characterized in that: The steps to train a feature learning network based on the ViT model include: Randomly initialize the network model's weight W1, weight decay coefficient WD1, and cross entropy loss function L CE , obtaining a first training round corresponding to the feature learning network based on the ViT model, wherein the first training round is from t=1 to t=T1; From a preset dataset Sample each pair of input x and true label y, propagate forward through the network, and calculate the model's predicted value y hat =f(x); Using the cross entropy loss function L CE Calculate the model's predicted value y hat The loss between the true label y; The cross entropy loss function L is calculated by back propagation CE The gradient of the weight W1 is updated and expressed as: The above steps are repeated until the number of training times reaches t=T1 to complete the training of the feature learning network based on the ViT model.

4. The method for detecting heartburn in lettuce for greenhouse vegetables according to any one of claim 3, characterized in that: The step of training the classifier comprises: Initialize the classifier's weight W2, weight decay coefficient WD2, and category balance loss function L CB and the maximum norm value max_norm, obtain the second training round corresponding to the classifier, where, W2=W1, the second training round is from t=1 to t=T2; Freeze all model parameters except the classifier, so that the classification head weight of the burnt lettuce detection model becomes trainable; From the dataset Sample each pair of input x and true label y, propagate forward through the network, and calculate the model's predicted value y hat =f(x); Using the class balance loss function L CB According to the predicted value y hat The loss between the real label y and different categories is given different loss weights; The loss function L is balanced according to the category CB The weight W2 is updated, which is expressed as: The maximum norm value max_norm is used to limit the norm of the updated weight W2 to ensure that the norm of the weight W2 after each update does not exceed the set maximum threshold; The above steps are repeated until the number of training times reaches t=T2 to complete the training of the classifier.

5. The method for detecting heartburn in lettuce for greenhouse vegetables according to claim 4, characterized in that: In the class balance loss function L CB A weight factor α is introduced to weight the loss according to the sample size of the category. The weight factor α is expressed as: Among them, β represents the hyperparameter that controls the distribution of category samples, and its value is less than 1, n represents the number of samples in the current category, N represents the total number of categories, and the weight factor α is used to give more attention and weight to a category when the number of samples in a certain category is small.

6. The method for detecting heartburn in lettuce for greenhouse vegetables according to claim 4, characterized in that: In the class balance loss function L CB A classification difficulty factor m is introduced, and the classification difficulty factor m is expressed as: m=exp(-γ[y×y hat +log(1+exp(-y hat ))]); Among them, γ is a hyperparameter used to adjust the sensitivity of the model to difficult-to-classify samples, and y represents the true label.

7. The method for detecting heartburn in lettuce for greenhouse vegetables according to claim 6, characterized in that: The number of the first Transformer encoding layers is 12, wherein each first Transformer encoding layer includes 12 multi-head attention encoding layers.

8. A lettuce heartburn detection device for greenhouse vegetables, characterized in that: include: a detection model construction module, configured to respond to a lettuce heartburn detection instruction for facility vegetables, obtain a lettuce image to be detected containing lettuce heartburn, and input the lettuce image to be detected into a heartburn lettuce detection model that has been trained to a convergence state, wherein the heartburn lettuce detection model is constructed by a feature learning network based on a ViT model followed by a classifier; A position matrix determination module is configured to divide the lettuce image to be detected into a plurality of image blocks in the feature learning network based on the ViT model, map the plurality of image blocks to a high-dimensional embedding space through a linear embedding layer, add a position matrix representing the position of the image block in the lettuce image to be detected to each image block, and initialize a classification matrix; An image block association module is configured to pass through multiple first Transformer encoding layers so that each attention head can learn the relationship between different image blocks to capture the correlation between different regions in the lettuce image to be detected, wherein each first Transformer encoding layer includes multiple multi-head attention encoding layers; a comprehensive image representation determination module, configured to perform matrix multiplication fusion of the multi-head attention weights from different first Transformer coding layers to determine a comprehensive image representation, and to screen out a plurality of image block matrices with maximum discriminability for classification according to the multiple attention head weights of each first Transformer coding layer; The heartburn lettuce detection module is configured to input the image block matrix with the maximum discriminability into a second Transformer encoding layer to further extract deep features to generate a global feature vector, and input the global feature vector into the classifier to determine that the lettuce image to be detected is heartburn lettuce, so as to complete the heartburn detection of lettuce for facility vegetables.

9. An electronic device, comprising a central processing unit and a memory, characterized in that: The central processing unit is used to call and run the computer program stored in the memory to execute the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: It stores a computer program implemented according to the method described in any one of claims 1 to 7 in the form of computer-readable instructions, and when the computer program is called and executed by a computer, the steps included in the corresponding method are executed.

Images