Rice ear and leaf width feature extraction method and system, electronic equipment and medium

Abstract

The invention discloses a rice spike and leaf width feature extraction method and system, electronic equipment and a medium, and the method comprises the steps: collecting a plurality of images containing rice spikes and rice leaves, and constructing a rice spike data set based on the plurality of images containing rice spikes and rice leaves; the rice spike data set is adopted to train a feature extraction network model comprising a pixel-level feature extraction network and a knowledge distillation network, a trained feature extraction network model is obtained, and the pixel-level feature extraction network is used for extracting initial features of rice spikes and leaf widths in the rice spike data set; and performing feature extraction on the rice image by using the trained feature extraction network model to obtain a rice spike target feature and a leaf width target feature in the rice image. The accuracy of feature extraction of the rice panicle and leaf width can be improved.

Description

Technical Field

[0001] The present application relates to the field of machine vision technology, and in particular to a method, system, electronic equipment and medium for extracting features of rice ears and leaf widths. Background Art

[0002] Most of the current rice ear feature extraction methods rely on manually annotated data, but the data annotation process of such methods is often cumbersome and time-consuming, and consumes a lot of manpower and resources. In addition, since rice ears are often mixed with weeds in rice ear images, data annotation cannot be refined. At the same time, the existing network cannot capture global context information due to the limitations of the receptive field.

[0003] Therefore, the current rice ear feature extraction method is not accurate enough in identifying the labels of image pixels, which makes it difficult to accurately extract the rice ear features, and the accuracy of feature extraction of rice ears and leaf width is relatively low. Summary of the invention

[0004] The present application aims to propose a method, system, electronic device and medium for extracting features of rice ears and leaf width, which can improve the accuracy of feature extraction of rice ears and leaf width.

[0005] In a first aspect, an embodiment of the present application provides a method for extracting features of rice ears and leaf widths, the method comprising:

[0006] Collecting a plurality of images including rice ears and rice leaves, and constructing a rice ear dataset based on the plurality of images including rice ears and rice leaves;

[0007] The rice ear dataset is used to train a feature extraction network model including a pixel-level feature extraction network and a knowledge distillation network to obtain a trained feature extraction network model, wherein the pixel-level feature extraction network is used to extract initial features of rice ears and leaf width in the rice ear dataset; so that the trained feature extraction network model can be used to perform feature extraction on rice images to obtain rice ear target features and leaf width target features in the rice images.

[0008] Compared with the prior art, the first aspect of the present application has the following beneficial effects:

[0009] The method collects multiple images containing rice ears and rice leaves, and constructs a rice ear dataset based on the multiple images containing rice ears and rice leaves; uses the rice ear dataset to train a feature extraction network model containing a pixel-level feature extraction network and a knowledge distillation network to obtain a trained feature extraction network model, and the pixel-level feature extraction network is used to extract the initial features of the rice ears and leaf width in the rice ear dataset; so as to use the trained feature extraction network model to extract features of the rice image, and obtain the target features of the rice ears and the target features of the leaf width in the rice image. In this way, the initial features of the rice ears and leaf width in the image are extracted by the pixel-level feature extraction network, and the extracted initial features are further extracted by the knowledge distillation network, which can improve the accuracy of the feature extraction of the rice ears and leaf width.

[0010] In some embodiments, the rice ear dataset is used to train a feature extraction network model including a pixel-level feature extraction network and a knowledge distillation network to obtain a trained feature extraction network model, including:

[0011] Performing global cropping and local cropping on the rice ear dataset to obtain a global rice ear dataset and a local rice ear dataset;

[0012] The pixel-level feature extraction network is trained by using the global rice ear dataset and the local rice ear dataset, and features are extracted from images in the global rice ear dataset and the local rice ear dataset by using the pixel-level feature extraction network to obtain a first rice ear extraction feature and a first leaf width extraction feature corresponding to the global rice ear dataset, and obtain a second rice ear extraction feature and a second leaf width extraction feature corresponding to the local rice ear dataset;

[0013] The first rice ear extraction feature, the first leaf width extraction feature, the second rice ear extraction feature and the second leaf width extraction feature are used to train the knowledge distillation network until the training of the knowledge distillation network is completed to obtain a trained feature extraction network model.

[0014] In some embodiments, the pixel-level feature extraction network includes a local transformer encoder and a global transformer encoder, the local transformer encoder has the same structure as the global transformer encoder, and the pixel-level feature extraction network is used to extract features from the images in the global rice ear dataset and the local rice ear dataset to obtain a first rice ear extraction feature and a first leaf width extraction feature corresponding to the global rice ear dataset, and to obtain a second rice ear extraction feature and a second leaf width extraction feature corresponding to the local rice ear dataset, including:

[0015] Dividing each image in the global rice ear dataset into a plurality of first image blocks, dividing each of the first image blocks into a plurality of first pixel blocks, and dividing each image in the local rice ear dataset into a plurality of second image blocks, dividing each of the second image blocks into a plurality of second pixel blocks;

[0016] Performing linear projection and position coding on each of the first pixel blocks to obtain a coded first pixel block, and performing linear projection and position coding on each of the second pixel blocks to obtain a coded second pixel block;

[0017] Inputting the encoded first pixel block into the local transformer encoder for feature extraction to obtain first pixel block features, and inputting the encoded second pixel block into the local transformer encoder for feature extraction to obtain second pixel block features;

[0018] Embedding the first pixel block feature into the plurality of first image blocks through a linear transformation layer to obtain a first embedded image block, and embedding the second pixel block feature into the plurality of second image blocks through a linear transformation layer to obtain a second embedded image block;

[0019] The first embedded block is input into the global transformer encoder for feature extraction to obtain a first rice ear extraction feature and a first leaf width extraction feature corresponding to the global rice ear dataset, and the second embedded block is input into the global transformer encoder for feature extraction to obtain a second rice ear extraction feature and a second leaf width extraction feature corresponding to the local rice ear dataset.

[0020] In some embodiments, the knowledge distillation network includes a student network and a teacher network, the student network and the teacher network have the same structure, and the knowledge distillation network is trained using the first rice ear extraction feature, the first leaf width extraction feature, the second rice ear extraction feature, and the second leaf width extraction feature until the knowledge distillation network is trained to obtain a trained feature extraction network model, including:

[0021] Inputting the first rice ear extraction feature, the first leaf width extraction feature, the second rice ear extraction feature and the second leaf width extraction feature into the student network to obtain a first feature distribution output by the student network;

[0022] Inputting the first rice ear extraction feature and the first leaf width extraction feature into the teacher network to obtain a second feature distribution output by the teacher network;

[0023] Calculating a cross entropy loss based on the first feature distribution and the second feature distribution;

[0024] According to the cross entropy loss, a gradient descent strategy is adopted to update the weight parameters of the student network;

[0025] The exponential moving average of the weight parameters of the student network is calculated, and the weight parameters of the teacher network are updated by the exponential moving average until a preset convergence condition is reached, and the knowledge distillation network is trained to obtain a trained feature extraction network model.

[0026] In some embodiments, inputting the first rice ear extraction feature, the first leaf width extraction feature, the second rice ear extraction feature, and the second leaf width extraction feature into the student network to obtain a first feature distribution output by the student network includes:

[0027]

[0028] Wherein, x represents any one of the first rice ear extraction feature, the first leaf width extraction feature, the second rice ear extraction feature, and the second leaf width extraction feature, P s (x) (i) represents the first feature distribution corresponding to the i-th feature output by the student network, represents the original output value of the student network corresponding to the i-th feature, τ s represents the temperature parameter of the student network, K represents the total number of features, exp(·) represents the exponential function, θ s represents the weight parameters of the student network.

[0029] In some embodiments, inputting the first rice ear extraction feature and the first leaf width extraction feature into the teacher network to obtain a second feature distribution output by the teacher network includes:

[0030]

[0031] Wherein, x′ represents any one of the first rice ear extraction feature and the first leaf width extraction feature, P t (x′) (i) represents the second feature distribution corresponding to the i-th feature output by the teacher network, represents the original output value of the teacher network corresponding to the i-th feature, τ t represents the temperature parameter of the teacher network, K represents the total number of features, exp(·) represents the exponential function, θ t represents the weight parameters of the teacher network.

[0032] In some implementations, calculating the cross entropy loss based on the first feature distribution and the second feature distribution includes:

[0033]

[0034] Among them, θ s represents the weight parameter of the student network, x′ represents any one of the first rice ear extraction feature and the first leaf width extraction feature, represents the first rice ear extraction feature, represents the first leaf width extraction feature, x represents any one of the first rice ear extraction feature, the first leaf width extraction feature, the second rice ear extraction feature and the second leaf width extraction feature, V represents the set of the first rice ear extraction feature, the first leaf width extraction feature, the second rice ear extraction feature and the second leaf width extraction feature, P t (x′) (i) represents the second feature distribution corresponding to the i-th feature output by the teacher network, P s (x) (i) represents the first feature distribution corresponding to the i-th feature output by the student network, and H(·) represents the cross entropy loss function.

[0035] In a second aspect, the present application also provides a feature extraction system for rice ears and leaf widths, the system comprising:

[0036] a data acquisition unit, configured to acquire a plurality of images containing rice ears and rice leaves, and construct a rice ear dataset based on the plurality of images containing rice ears and rice leaves;

[0037] A feature extraction unit is used to train a feature extraction network model including a pixel-level feature extraction network and a knowledge distillation network using the rice ear dataset to obtain a trained feature extraction network model, wherein the pixel-level feature extraction network is used to extract initial features of rice ears and leaf widths in the rice ear dataset; so as to use the trained feature extraction network model to perform feature extraction on rice images to obtain rice ear target features and leaf width target features in the rice images.

[0038] In a third aspect, an embodiment of the present application also provides an electronic device comprising at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions executable by the at least one control processor, and the instructions are executed by the at least one control processor so that the at least one control processor can execute a feature extraction method for rice ears and leaf width as described above.

[0039] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the method for extracting features of rice ears and leaf width as described above.

[0040] It can be understood that the beneficial effects of the second to fourth aspects compared with the related art are the same as the beneficial effects of the first aspect compared with the related art. Please refer to the relevant description in the first aspect, and no further details will be given here. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0042] Figure 1 It is a schematic flow chart of an embodiment of a method for extracting features of rice ears and leaf widths provided in the present application;

[0043] Figure 2 It is a schematic diagram of the structure of a pixel-level feature extraction network in the best embodiment of the feature extraction method for rice ear and leaf width provided by the present application;

[0044] Figure 3 : is a schematic diagram of the structure of the knowledge distillation network in the best embodiment of the method for extracting the features of rice ears and leaf widths provided in the present application;

[0045] Figure 4 It is a structural schematic diagram of a local multi-head self-attention module in the best embodiment of the feature extraction method of rice ears and leaf width provided by the present application;

[0046] Figure 5 It is a structural schematic diagram of a global multi-head self-attention module in the best embodiment of the feature extraction method of rice ears and leaf width provided by the present application;

[0047] Figure 6 It is a structural schematic diagram of an embodiment of a feature extraction system for rice ears and leaf widths provided in the present application. DETAILED DESCRIPTION

[0048] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as limiting the present application.

[0049] In the description of this application, if there is a description of first, second, etc., it is only for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0050] In the description of the present application, it should be understood that the descriptions involving orientation, such as the orientation or positional relationship indicated as up, down, etc., are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.

[0051] In the description of this application, it should be noted that, unless otherwise clearly defined, terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meaning of the above terms in this application based on the specific content of the technical solution.

[0052] Most of the current rice ear feature extraction methods rely on manually annotated data, but the data annotation process of such methods is often cumbersome and time-consuming, and consumes a lot of manpower and resources. In addition, since rice ears are often mixed with weeds in rice ear images, data annotation cannot be refined. At the same time, the existing network cannot capture global context information due to the limitations of the receptive field.

[0053] Therefore, the current rice ear feature extraction method is not accurate enough in identifying the labels of image pixels, which makes it difficult to accurately extract the rice ear features, and the accuracy of feature extraction of rice ears and leaf width is relatively low.

[0054] In order to solve the problem of low accuracy in feature extraction of rice ears and leaf width, the present application proposes a method, system, electronic device and medium for feature extraction of rice ears and leaf width.

[0055] Reference Figure 1 The present application embodiment provides a method for extracting features of rice ears and leaf widths, the method comprising the following steps:

[0056] Step S100, collecting a plurality of images containing rice ears and rice leaves, and constructing a rice ear dataset based on the plurality of images containing rice ears and rice leaves;

[0057] Step S200, using the rice ear dataset to train the feature extraction network model including the pixel-level feature extraction network and the knowledge distillation network to obtain a trained feature extraction network model, the pixel-level feature extraction network is used to extract the initial features of the rice ears and leaf width in the rice ear dataset; so as to use the trained feature extraction network model to extract features of the rice image to obtain the target features of the rice ears and the target features of the leaf width in the rice image.

[0058] In this embodiment, a rice ear dataset is constructed by collecting multiple images containing rice ears and rice leaves, and based on the multiple images containing rice ears and rice leaves; the rice ear dataset is used to train a feature extraction network model containing a pixel-level feature extraction network and a knowledge distillation network to obtain a trained feature extraction network model, and the pixel-level feature extraction network is used to extract the initial features of the rice ears and leaf width in the rice ear dataset; so that the trained feature extraction network model is used to extract features of the rice image to obtain the target features of the rice ears and the target features of the leaf width in the rice image. In this way, the initial features of the rice ears and leaf width in the image are extracted by the pixel-level feature extraction network, and the extracted initial features are further extracted by the knowledge distillation network, which can improve the accuracy of the feature extraction of the rice ears and leaf width.

[0059] The above-mentioned method of collecting multiple images containing rice ears and rice leaves and constructing a rice ear dataset based on the multiple images containing rice ears and rice leaves may be a method of collecting multiple images containing rice ears and rice leaves from different angles using an image acquisition device, and then deleting blurred, low-resolution, repeated, irrelevant, etc. rice ear images to construct a clean rice ear dataset.

[0060] The above-mentioned use of the rice ear dataset to train the feature extraction network model including the pixel-level feature extraction network and the knowledge distillation network to obtain the trained feature extraction network model can be performed by first processing the rice ear dataset, and then using the processed rice ear dataset to train the feature extraction network model including the pixel-level feature extraction network and the knowledge distillation network to obtain the trained feature extraction network model.

[0061] The above initial features can be features obtained by extracting features of rice ears and leaf widths in a rice ear dataset through a pixel-level feature extraction network.

[0062] In some embodiments, a feature extraction network model including a pixel-level feature extraction network and a knowledge distillation network is trained using a rice ear dataset to obtain a trained feature extraction network model, including:

[0063] Perform global cropping and local cropping on the rice ear dataset to obtain a global rice ear dataset and a local rice ear dataset;

[0064] A pixel-level feature extraction network is trained using a global rice ear dataset and a local rice ear dataset, and features are extracted from images in the global rice ear dataset and the local rice ear dataset through the pixel-level feature extraction network to obtain a first rice ear extraction feature and a first leaf width extraction feature corresponding to the global rice ear dataset, and a second rice ear extraction feature and a second leaf width extraction feature corresponding to the local rice ear dataset;

[0065] The knowledge distillation network is trained using the first rice ear extraction feature, the first leaf width extraction feature, the second rice ear extraction feature and the second leaf width extraction feature until the knowledge distillation network training is completed to obtain a trained feature extraction network model.

[0066] In this embodiment, by globally cropping and locally cropping the rice ear dataset, the vision can be enriched, thereby enhancing the generalization ability of the feature extraction network model. The knowledge distillation network is trained by obtaining the first rice ear extraction feature and the first leaf width extraction feature corresponding to the global rice ear dataset, and the second rice ear extraction feature and the second leaf width extraction feature corresponding to the local rice ear dataset. The similarity between pairs of images from different perspectives can be maximized through unsupervised self-distillation, and rice ear and leaf width features containing obvious semantic information can be extracted. Pixel-level feature labels are learned to achieve accurate extraction of rice ear and leaf width features of the image.

[0067] In some embodiments, the pixel-level feature extraction network includes a local transformer encoder and a global transformer encoder, the local transformer encoder has the same structure as the global transformer encoder, and features are extracted from images in the global rice ear dataset and the local rice ear dataset through the pixel-level feature extraction network to obtain a first rice ear extraction feature and a first leaf width extraction feature corresponding to the global rice ear dataset, and obtain a second rice ear extraction feature and a second leaf width extraction feature corresponding to the local rice ear dataset, including:

[0068] Divide each image in the global rice ear dataset into a plurality of first image blocks, divide each first image block into a plurality of first pixel blocks, and divide each image in the local rice ear dataset into a plurality of second image blocks, divide each second image block into a plurality of second pixel blocks;

[0069] Performing linear projection and position coding on each first pixel block to obtain a coded first pixel block, and performing linear projection and position coding on each second pixel block to obtain a coded second pixel block;

[0070] Inputting the encoded first pixel block into the local transformer encoder for feature extraction to obtain first pixel block features, and inputting the encoded second pixel block into the local transformer encoder for feature extraction to obtain second pixel block features;

[0071] Embedding the first pixel block feature into a plurality of first image blocks through a linear transformation layer to obtain a first embedded image block, and embedding the second pixel block feature into a plurality of second image blocks through a linear transformation layer to obtain a second embedded image block;

[0072] The first embedded block is input into the global transformer encoder for feature extraction to obtain the first rice ear extraction feature and the first leaf width extraction feature corresponding to the global rice ear dataset, and the second embedded block is input into the global transformer encoder for feature extraction to obtain the second rice ear extraction feature and the second leaf width extraction feature corresponding to the local rice ear dataset.

[0073] In this embodiment, each image in the global rice ear data set is divided into multiple first blocks, each first block is divided into multiple first pixel blocks, and each image in the local rice ear data set is divided into multiple second blocks, each second block is divided into multiple second pixel blocks, and then the local transformer encoder and the global transformer encoder are used to process the pixel blocks and the blocks respectively, and the local transformer encoder is used to analyze the similarity between the pixels inside the small image blocks, and the pixel points are labeled and classified, so that the detail information such as the edge of the rice ear can be accurately extracted, and then the global transformer encoder is used to analyze the similarity between the small image blocks, and the small blocks are classified, so as to further realize the extraction of the shape and position relationship of the rice ear and other information, and the local features and global features of the image can be extracted, and the visual information can be completely extracted, so as to lay a good data foundation for the later training of the knowledge distillation network.

[0074] In some embodiments, the knowledge distillation network includes a student network and a teacher network, the student network and the teacher network have the same structure, and the knowledge distillation network is trained using the first rice ear extraction feature, the first leaf width extraction feature, the second rice ear extraction feature, and the second leaf width extraction feature until the knowledge distillation network training is completed, and a trained feature extraction network model is obtained, including:

[0075] Inputting the first rice ear extraction feature, the first leaf width extraction feature, the second rice ear extraction feature and the second leaf width extraction feature into the student network to obtain a first feature distribution output by the student network;

[0076] Inputting the first rice ear extraction feature and the first leaf width extraction feature into the teacher network to obtain a second feature distribution output by the teacher network;

[0077] Calculate the cross entropy loss based on the first feature distribution and the second feature distribution;

[0078] According to the cross entropy loss, the gradient descent strategy is used to update the weight parameters of the student network;

[0079] The exponential moving average of the weight parameters of the student network is calculated, and the weight parameters of the teacher network are updated by exponential moving average until the preset convergence condition is reached. After that, the knowledge distillation network is trained to obtain a trained feature extraction network model.

[0080] In this embodiment, the first rice ear extraction feature, the first leaf width extraction feature, the second rice ear extraction feature and the second leaf width extraction feature are input into the student network to obtain the first feature distribution output by the student network; then the first rice ear extraction feature and the first leaf width extraction feature are input into the teacher network to obtain the second feature distribution output by the teacher network; then based on the first feature distribution and the second feature distribution, the cross entropy loss is calculated; according to the cross entropy loss, the weight parameters of the student network are updated using the gradient descent strategy; the exponential moving average of the weight parameters of the student network is calculated, and the weight parameters of the teacher network are updated by the exponential moving average until the preset convergence condition is reached, and the knowledge distillation network is trained to obtain a trained feature extraction network model. In this way, by comparing the extracted features generated by different views, and combining the rice ears and their surrounding environment information in different image regions, the rice ears and leaf width features are more accurately extracted; then a momentum-quantized teacher network is used to generate high-quality pseudo labels (target distribution of the student network) based on the global view, and the pseudo labels are used to supervise and guide the learning of the student network. In this way, it is achieved that the rice ear features and leaf width features can be accurately extracted without relying on artificially labeled data.

[0081] In some embodiments, the first rice ear extraction feature, the first leaf width extraction feature, the second rice ear extraction feature, and the second leaf width extraction feature are input into the student network to obtain a first feature distribution output by the student network, including:

[0082]

[0083] Wherein, x represents any one of the first rice ear extraction feature, the first leaf width extraction feature, the second rice ear extraction feature, and the second leaf width extraction feature, P s (x) (i) represents the first feature distribution corresponding to the i-th feature output by the student network, represents the original output value of the student network corresponding to the i-th feature, τ s represents the temperature parameter of the student network, K represents the total number of features, exp(·) represents the exponential function, θ s Represents the weight parameter of the student network.

[0084] In some embodiments, the first rice ear extraction feature and the first leaf width extraction feature are input into the teacher network to obtain a second feature distribution output by the teacher network, including:

[0085]

[0086] Where x′ represents any one of the first rice ear extraction feature and the first leaf width extraction feature, P t (x′) (i)represents the second feature distribution corresponding to the i-th feature output by the teacher network, represents the original output value of the teacher network corresponding to the i-th feature, τ t represents the temperature parameter of the teacher network, K represents the total number of features, exp(·) represents the exponential function, and θ t Represents the weight parameters of the teacher network.

[0087] In some implementations, calculating a cross entropy loss based on the first feature distribution and the second feature distribution includes:

[0088]

[0089] Among them, θ s represents the weight parameter of the student network, x′ represents any one of the first rice ear extraction feature and the first leaf width extraction feature, represents the first rice ear extraction feature, represents the first leaf width extraction feature, x represents any one of the first rice ear extraction feature, the first leaf width extraction feature, the second rice ear extraction feature and the second leaf width extraction feature, V represents the set of the first rice ear extraction feature, the first leaf width extraction feature, the second rice ear extraction feature and the second leaf width extraction feature, P t (x′) (i) represents the second feature distribution corresponding to the i-th feature output by the teacher network, P s (x) (i) represents the first feature distribution corresponding to the i-th feature output by the student network, and H(·) represents the cross entropy loss function.

[0090] To facilitate understanding by those skilled in the art, a set of best embodiments is provided below:

[0091] The existing rice ear and leaf width feature extraction methods have the following disadvantages:

[0092] 1. Most of the current rice ear feature extraction methods rely on manually labeled data, but the data labeling process of such methods is often cumbersome and time-consuming, and consumes a lot of manpower costs and resources.

[0093] Reason: Since supervised training is used, human efforts are required to label the data. However, there are often thousands or even tens of thousands of labeled data, which results in high human cost and resource consumption.

[0094] 2. Most of the current rice ear feature extraction methods are unable to extract explicit information for image semantic segmentation, and the label recognition of image pixels is not accurate enough.

[0095] Reason: In rice ear images, rice ears are often mixed with weeds, so data annotation cannot be refined. At the same time, the existing network cannot capture global context information due to the limitations of the receptive field, making it difficult to accurately extract rice ear features.

[0096] In order to solve the problems existing in the above-mentioned existing rice ear and leaf width feature extraction methods, this embodiment provides a self-supervised label-free rice ear feature extraction method, which specifically includes the following contents:

[0097] (1) Collect RGB images of rice ears.

[0098] This embodiment uses a handheld camera to obtain diversified rice ear images (ie, rice ear images) including images of different varieties, different growing environments, different lighting conditions, and different shooting angles.

[0099] (2) Construct a dataset.

[0100] Blurred, low-resolution, repeated, and irrelevant rice ear images were deleted to build a clean rice ear dataset.

[0101] (3) Data processing

[0102] To enrich the visual representation and enhance the generalization ability of the model, each rice ear image is cropped in different sizes, including 2 global crops (cropping 50-100% of the area on the input rice ear image and resizing it to 224) to obtain a global view, 8 local crops (cropping 5-50% of the area on the input rice ear image and resizing it to 96) to obtain a local view, and image enhancement operations are performed on all cropped images;

[0103] (4) Construct a pixel-level feature extraction network.

[0104] A pixel-level feature extraction network is established, and the model (i.e., the pixel-level feature extraction network of the present embodiment) is trained using the rice ear dataset processed in step (3). Each input cropped and enhanced rice ear image is divided into multiple image blocks, and the image blocks are processed based on two transformer encoders to extract information. The internal transformer block (i.e., the local transformer encoder) is used to analyze the similarity between the pixels inside the image blocks, classify the pixel points, and accurately extract detailed information such as the edge of the rice ear; the external transformer block (i.e., the global transformer encoder) analyzes the similarity between the image blocks, classifies the blocks, further realizes the extraction of information such as the shape and position relationship of the rice ear, and outputs the visual feature vector.

[0105] Reference Figure 2, which is a schematic diagram of the structure of the pixel-level feature extraction network. The neural network structure of the pixel-level feature extraction network 1 is as follows:

[0106] 1) Using two transformer encoders as the backbone, the input image is evenly divided into n blocks, each of which is further divided into m pixel blocks. The pixel block is input into the local transformer encoder (i.e., local transformer encoder), and the block is input into the global transformer encoder (i.e., global transformer encoder). The two transformer encoders have the same structure, including multi-head self-attention mechanism (MSA), residual connection and layer normalization (Add&Norm) and feed-forward neural network (FFN).

[0107] 2) First, the pixel block is encoded by linear projection and position encoding is added, and then input into the local transformer encoder. Then, through the multi-head self-attention mechanism, each pixel block is self-attentioned with other pixel blocks to extract the relevant information between pixel blocks and capture different information at different positions; then, it is normalized through the Add&Norm layer; finally, the visual features of the pixel block are further extracted through FFN and Add&Norm, so as to achieve accurate extraction of detailed information such as the edge and texture of rice ears.

[0108] 3) Create an embedding memory to store the sequence represented by the tile, project the pixel block features to the tile embedding space through the linear transformation layer, and then add it to the tile, input the embedding of each tile into the global transformer encoder for further feature extraction. Since the structure of the global transformer encoder is the same as that of the local transformer encoder, the feature extraction process of the global transformer encoder in step 3) is the same as that of the local transformer encoder in step 2), further realizing the extraction of information such as the shape and position relationship of the rice ear, and outputting the tile feature vector.

[0109] (5) Build a knowledge distillation network.

[0110] A knowledge distillation network is established, and the model (i.e., the knowledge distillation network of this embodiment) is trained using the visual features output by step (4) (i.e., the features output by the pixel-level feature extraction network). A method based on knowledge distillation is adopted to more accurately extract the features of rice ears and leaf width by comparing the feature vectors of rice ears images generated based on different views, and combining the information of rice ears and their surroundings in different image regions. A momentum-based teacher network is used to generate high-quality pseudo labels (target distribution of the student network) based on the global view, and the pseudo labels are used to supervise and guide the learning of the student network.

[0111] Reference Figure 3 , is a schematic diagram of the structure of the knowledge distillation network. The neural network structure of the knowledge distillation network is as follows:

[0112] 1) The knowledge distillation network is divided into a student network and a teacher network. The two networks have the same structure, both of which contain 4 blocks with the same structure, 3 downsampling modules and a projection head. The blocks and downsampling modules are alternately connected. Each block consists of a GTG module similar to a convolutional neural network, alternating local multi-head self-attention modules (i.e., local MSA) and global multi-head self-attention modules (i.e., global MSA). The GTG module includes layer normalization, convolutional layers and Fused MBConv (Fused Mobile Inverted Residual Bottleneck). Fused MBConv is a network module in the field of deep learning, mainly used to build efficient convolutional neural networks. In the local multi-head self-attention module, local features are extracted by calculating the correlation between adjacent rice ear blocks; in the global multi-head self-attention module, the global context information (i.e., the overall features and relationships between rice ears and their surroundings) is used to model long-term dependencies, and global features are extracted through GTG and interact with local features. Each downsampling module will reduce the spatial resolution of the feature map space by 2 times and increase the number of channels by 2 times. The global context information of different image regions is combined to extract rice ear and leaf width features.

[0113] The local multi-head self-attention module and the global multi-head self-attention module are specifically:

[0114] Both the local multi-head self-attention module and the global multi-head self-attention module use the multi-head attention mechanism. The only difference between the two is the source of the query: the former relies on linear transformation and embedding of each rice ear tile to generate a series of queries: Q, while the latter relies on the GTG structure to generate a query: global Q. In the local self-attention module, each rice ear tile will generate its own Q, K, V matrix and perform attention calculations independently. In the global self-attention module, the global Q will perform attention calculations with the K and V matrices of all rice ear tiles to output global attention. Figure 4 , is a schematic diagram of the structure of the local multi-head self-attention module, refer to Figure 5 , which is a structural diagram of the global multi-head self-attention module.

[0115] 2) All the cropped rice ears in step (3) (including the global cropping and local cropping in step (3)) are input into the student network through the block feature vector obtained in step (4) to obtain the feature distribution; only the global cropped rice ears in step (3) (including only the global cropping in step (3)) are input into the teacher network through the block feature vector obtained in step (4) to obtain the feature distribution. The calculation formula of the feature distribution is as follows:

[0116]

[0117] Wherein, x represents any one of the first rice ear extraction feature, the first leaf width extraction feature, the second rice ear extraction feature, and the second leaf width extraction feature, P s (x) (i) represents the first feature distribution corresponding to the i-th feature output by the student network, that is, represents the normalized value of the i-th feature of the student network for the input x. represents the original output value of the student network corresponding to the i-th feature, K represents the total number of features, exp(·) represents the exponential function, and θ s represents the weight parameter of the student network, x′ represents any one of the first rice ear extraction feature and the first leaf width extraction feature, P t (x′) (i) represents the second feature distribution corresponding to the i-th feature output by the teacher network, that is, represents the normalized value of the i-th feature of the teacher network for the input x′, represents the original output value of the teacher network corresponding to the i-th feature, τ s represents the temperature parameter of the student network, τ t represents the temperature parameter of the teacher network, which is used to control the output distribution curve, θ t Represents the weight parameters of the teacher network.

[0118] 3) Calculate the loss of the probability distribution between the feature distribution of the student network and the teacher network, and then update the weight parameters of the student network according to the calculated loss gradient. The loss function formula is as follows:

[0119]

[0120] in, represents the first rice ear extraction feature, represents the first leaf width extraction feature, and is the feature vector of the tile in the global rice ear cropped image (i.e., the image in the global rice ear dataset), V represents the set of the first rice ear extraction feature, the first leaf width extraction feature, the second rice ear extraction feature, and the second leaf width extraction feature, i.e., the tile feature vector in all rice ear cropped images (i.e., including the images in the global rice ear dataset and the images in the local rice ear dataset), and H(·) represents the cross entropy loss function.

[0121] The weight parameters of the teacher network are updated by the exponential moving average (EMA) of the student network weights; that is:

[0122] θ t ←λθ t +(1-λ)θ s

[0123] Among them, λ follows the cosine strategy and takes values ​​from 0.996 to 1.

[0124] 4) By continuously updating the weight parameters of the student network and the teacher network, the student network and the teacher network are optimized to output the same result trend, thereby improving the accuracy of rice ear and leaf width feature extraction through self-supervision.

[0125] (6) A feature extraction network model is obtained by constructing a pixel-level feature extraction network and a knowledge distillation network. After the pixel-level feature extraction network and the knowledge distillation network are trained, a trained feature extraction network model can be obtained. By using the trained feature extraction network model, feature extraction is performed on rice images (i.e., new images collected containing rice ears and rice leaves), and target features of rice ears and target features of leaf width can be obtained.

[0126] Compared with the prior art, the technical solution of this embodiment has the following advantages:

[0127] 1. Provide a self-supervised unlabeled rice ear feature extraction method, that is, generate high-quality pseudo labels through a momentum-based teacher network to guide the learning process of the student network, continuously optimize the network structure, and realize self-supervised unlabeled extraction of rice ear and leaf width features. By self-generating pseudo labels to supervise the training network, accurate extraction of rice ear and leaf width features is achieved without labeled data.

[0128] 2. By extracting global and local features of the image, the visual information is completely extracted. The similarity between image pairs from different perspectives is maximized by comparing the similarity between image pixels through unsupervised self-distillation. The rice ear and leaf width features containing obvious semantic information are extracted, and pixel-level feature labels are learned to achieve accurate extraction of the rice ear and leaf width features of the image.

[0129] Reference Figure 6 The embodiment of the present application further provides a feature extraction system for rice ears and leaf widths, the system comprising a data acquisition unit 100 and a feature extraction unit 200, wherein:

[0130] The data acquisition unit 100 is used to acquire a plurality of images including rice ears and rice leaves, and construct a rice ear dataset based on the plurality of images including rice ears and rice leaves;

[0131] The feature extraction unit 200 is used to train a feature extraction network model including a pixel-level feature extraction network and a knowledge distillation network using a rice ear dataset to obtain a trained feature extraction network model. The pixel-level feature extraction network is used to extract initial features of rice ears and leaf widths in the rice ear dataset; so that the trained feature extraction network model can be used to extract features of rice images to obtain target features of rice ears and leaf widths in the rice images.

[0132] It should be noted that since the feature extraction system of rice ears and leaf width in this embodiment and the feature extraction method of rice ears and leaf width mentioned above are based on the same inventive concept, the corresponding contents in the method embodiment are also applicable to the system embodiment and will not be described in detail here.

[0133] An embodiment of the present application also provides an electronic device, comprising: at least one control processor and a memory for communicating with the at least one control processor.

[0134] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0135] The non-transient software program and instructions required to implement the feature extraction method of a rice ear and leaf width of the above embodiment are stored in the memory. When executed by the processor, the feature extraction method of a rice ear and leaf width of the above embodiment is executed, for example, the above described method is executed. Figure 1 The method comprises steps S100 to S200.

[0136] The system embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0137] The present application also provides a computer-readable storage medium storing computer-executable instructions, which are executed by one or more control processors to enable the one or more control processors to execute a method for extracting rice ears and leaf width features in the above method embodiment, for example, to execute the above described Figure 1 The functions of method steps S100 to S200 in the method.

[0138] It will be appreciated by those skilled in the art that all or some of the steps and systems in the disclosed method above may be implemented as software, firmware, hardware and appropriate combinations thereof. Some physical components or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that may be used to store desired information and may be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0139] The above is a specific description of the preferred implementation of the embodiments of the present application, but the embodiments of the present application are not limited to the above-mentioned implementation methods. Technical personnel familiar with the field can also make various equivalent modifications or substitutions without violating the spirit of the embodiments of the present application. These equivalent modifications or substitutions are all included in the scope defined by the claims of the embodiments of the present application.

Claims

1. A method for extracting features of rice ears and leaf widths, characterized in that: The method comprises: Collecting a plurality of images including rice ears and rice leaves, and constructing a rice ear dataset based on the plurality of images including rice ears and rice leaves; The rice ear dataset is used to train a feature extraction network model including a pixel-level feature extraction network and a knowledge distillation network to obtain a trained feature extraction network model, wherein the pixel-level feature extraction network is used to extract initial features of rice ears and leaf width in the rice ear dataset; so that the trained feature extraction network model can be used to perform feature extraction on rice images to obtain rice ear target features and leaf width target features in the rice images.

2. The method for extracting the features of rice ears and leaf width according to claim 1, characterized in that: The method of using the rice ear dataset to train a feature extraction network model including a pixel-level feature extraction network and a knowledge distillation network to obtain a trained feature extraction network model includes: Performing global cropping and local cropping on the rice ear dataset to obtain a global rice ear dataset and a local rice ear dataset; The pixel-level feature extraction network is trained by using the global rice ear dataset and the local rice ear dataset, and features are extracted from images in the global rice ear dataset and the local rice ear dataset by using the pixel-level feature extraction network to obtain a first rice ear extraction feature and a first leaf width extraction feature corresponding to the global rice ear dataset, and obtain a second rice ear extraction feature and a second leaf width extraction feature corresponding to the local rice ear dataset; The first rice ear extraction feature, the first leaf width extraction feature, the second rice ear extraction feature and the second leaf width extraction feature are used to train the knowledge distillation network until the training of the knowledge distillation network is completed to obtain a trained feature extraction network model.

3. The method for extracting the features of rice ears and leaf width according to claim 2, characterized in that: The pixel-level feature extraction network includes a local transformer encoder and a global transformer encoder, the local transformer encoder has the same structure as the global transformer encoder, and the pixel-level feature extraction network is used to extract features from the images in the global rice ear dataset and the local rice ear dataset, to obtain a first rice ear extraction feature and a first leaf width extraction feature corresponding to the global rice ear dataset, and to obtain a second rice ear extraction feature and a second leaf width extraction feature corresponding to the local rice ear dataset, including: Dividing each image in the global rice ear dataset into a plurality of first image blocks, dividing each of the first image blocks into a plurality of first pixel blocks, and dividing each image in the local rice ear dataset into a plurality of second image blocks, dividing each of the second image blocks into a plurality of second pixel blocks; Performing linear projection and position coding on each of the first pixel blocks to obtain a coded first pixel block, and performing linear projection and position coding on each of the second pixel blocks to obtain a coded second pixel block; Inputting the encoded first pixel block into the local transformer encoder for feature extraction to obtain first pixel block features, and inputting the encoded second pixel block into the local transformer encoder for feature extraction to obtain second pixel block features; Embedding the first pixel block feature into the plurality of first image blocks through a linear transformation layer to obtain a first embedded image block, and embedding the second pixel block feature into the plurality of second image blocks through a linear transformation layer to obtain a second embedded image block; The first embedded block is input into the global transformer encoder for feature extraction to obtain a first rice ear extraction feature and a first leaf width extraction feature corresponding to the global rice ear dataset, and the second embedded block is input into the global transformer encoder for feature extraction to obtain a second rice ear extraction feature and a second leaf width extraction feature corresponding to the local rice ear dataset.

4. The method for extracting the features of rice ears and leaf width according to claim 2, characterized in that: The knowledge distillation network includes a student network and a teacher network, the student network and the teacher network have the same structure, the first rice ear extraction feature, the first leaf width extraction feature, the second rice ear extraction feature and the second leaf width extraction feature are used to train the knowledge distillation network until the knowledge distillation network training is completed, and a trained feature extraction network model is obtained, including: Inputting the first rice ear extraction feature, the first leaf width extraction feature, the second rice ear extraction feature and the second leaf width extraction feature into the student network to obtain a first feature distribution output by the student network; Inputting the first rice ear extraction feature and the first leaf width extraction feature into the teacher network to obtain a second feature distribution output by the teacher network; Calculating a cross entropy loss based on the first feature distribution and the second feature distribution; According to the cross entropy loss, a gradient descent strategy is adopted to update the weight parameters of the student network; The exponential moving average of the weight parameters of the student network is calculated, and the weight parameters of the teacher network are updated by the exponential moving average until a preset convergence condition is reached, and the knowledge distillation network is trained to obtain a trained feature extraction network model.

5. The method for extracting the features of rice ears and leaf width according to claim 4, characterized in that: The inputting the first rice ear extraction feature, the first leaf width extraction feature, the second rice ear extraction feature and the second leaf width extraction feature into the student network to obtain a first feature distribution output by the student network comprises: Wherein, x represents any one of the first rice ear extraction feature, the first leaf width extraction feature, the second rice ear extraction feature, and the second leaf width extraction feature, P s (x) (i) represents the first feature distribution corresponding to the i-th feature output by the student network, gθ s (x) (i) represents the original output value of the student network corresponding to the i-th feature, τs represents the temperature parameter of the student network, K represents the total number of features, exp(·) represents the exponential function, θ s represents the weight parameters of the student network.

6. The method for extracting the features of rice ears and leaf width according to claim 4, characterized in that: The step of inputting the first rice ear extraction feature and the first leaf width extraction feature into the teacher network to obtain a second feature distribution output by the teacher network includes: Wherein, x′ represents any one of the first rice ear extraction feature and the first leaf width extraction feature, P t (x′) (i) represents the second feature distribution corresponding to the i-th feature output by the teacher network, represents the original output value of the teacher network corresponding to the i-th feature, τ t represents the temperature parameter of the teacher network, K represents the total number of features, exp(·) represents the exponential function, θ t represents the weight parameters of the teacher network.

7. The method for extracting the features of rice ears and leaf widths according to claim 4, characterized in that: The calculating the cross entropy loss based on the first feature distribution and the second feature distribution includes: Among them, θ s represents the weight parameter of the student network, x′ represents any one of the first rice ear extraction feature and the first leaf width extraction feature, represents the first rice ear extraction feature, represents the first leaf width extraction feature, x represents any one of the first rice ear extraction feature, the first leaf width extraction feature, the second rice ear extraction feature and the second leaf width extraction feature, V represents the set of the first rice ear extraction feature, the first leaf width extraction feature, the second rice ear extraction feature and the second leaf width extraction feature, P t (x′) (i) represents the second feature distribution corresponding to the i-th feature output by the teacher network, P s (x) (i) represents the first feature distribution corresponding to the i-th feature output by the student network, and H(·) represents the cross entropy loss function.

8. A feature extraction system for rice ear and leaf width, characterized in that: The system comprises: a data acquisition unit, configured to acquire a plurality of images containing rice ears and rice leaves, and construct a rice ear dataset based on the plurality of images containing rice ears and rice leaves; A feature extraction unit is used to train a feature extraction network model including a pixel-level feature extraction network and a knowledge distillation network using the rice ear dataset to obtain a trained feature extraction network model, wherein the pixel-level feature extraction network is used to extract initial features of rice ears and leaf widths in the rice ear dataset; so as to use the trained feature extraction network model to perform feature extraction on rice images to obtain rice ear target features and leaf width target features in the rice images.

9. An electronic device, characterized in that: It includes at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions that can be executed by the at least one control processor, and the instructions are executed by the at least one control processor so that the at least one control processor can execute the rice ear and leaf width feature extraction method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the method for extracting features of rice ears and leaf widths according to any one of claims 1 to 7.

Images