Training method for grain identification model, grain identification method, equipment and medium

Abstract

The invention discloses a training method, a grain recognition method, a device and a medium for a grain recognition model, and relates to the field of grain recognition. The training method for the grain recognition model includes: obtaining a plurality of grain images and text labels corresponding to each grain image for model training; constructing a grain recognition model to be trained, and extracting features of each grain image and the corresponding text label based on the grain recognition model to obtain image feature data and text feature data; determining a similarity score between the image feature data and the text feature data based on the grain recognition model; constructing a multimodal supervised contrast loss function, and determining a loss value between a similarity score and a reference score based on the multimodal supervised contrast loss function, and adjusting the parameters of the grain recognition model based on the loss value to iteratively train the model to obtain a trained grain recognition model. The grain recognition model of the invention can be widely used in various types of grain detection to improve detection efficiency.

Description

Technical Field

[0001] The present application relates to technical fields such as grain identification and model training, and in particular to a grain identification model training method, a grain identification method, a device and a medium. Background Art

[0002] Grain quality evaluation is an important technical support for grain production, procurement, storage and processing. Its evaluation indicators are of great significance in guiding procurement pricing and guiding the rational use of grains. Grain quality index inspection mainly includes physical property indicators (bulk density, thousand-grain weight, hardness and imperfect grains, etc.), chemical property indicators (protein, gluten, sediment, mold activity, etc.) and edible quality. In the field of grain quality inspection, physical property indicators are widely used, which requires rapid field inspection based on accurate results. Therefore, high-light spectrum detection or image detection is currently more commonly used to standardize sensory inspection indicators, reduce subjective errors in inspections, and improve inspection levels.

[0003] In recent years, deep learning technology represented by convolutional neural networks has developed rapidly and has shown great application potential in various fields. In terms of image detection technology, deep learning recognition models are currently mainly relied on to identify indicators such as sensory inspection of individual grains. These methods only use image encoders to extract image features and map them into logical vectors, which are then compared with labels in the form of one-hot (a text feature representation method) and cross entropy is used for learning optimization. One disadvantage of this is that all categories are mapped into one-hot vectors, without considering the text semantic information of each category, so the model needs to be retrained every time a grain category is added. This restricted form of supervision limits the versatility of the model. Summary of the invention

[0004] To this end, the purpose of the implementation mode of the present application is to propose a grain recognition model training method, grain recognition method, equipment, medium and computer program. The trained grain recognition model covers a variety of grain types, can be widely used in various types of grain detection tasks, and is more universal.

[0005] An embodiment of the present application provides a method for training a grain recognition model, the method comprising: obtaining a plurality of grain images for model training and a text label corresponding to each grain image; constructing a grain recognition model to be trained, and performing feature extraction on each grain image and the corresponding text label based on the grain recognition model to obtain image feature data and text feature data; determining a similarity score between the image feature data and the text feature data based on the grain recognition model; constructing a multimodal supervised contrast loss function, and determining a loss value between the similarity score and a reference score based on the multimodal supervised contrast loss function, and adjusting model parameters of the grain recognition model based on the loss value, so as to iteratively train the model to obtain a trained grain recognition model.

[0006] Exemplarily, the multimodal supervised contrast loss function includes a KL divergence multimodal supervised contrast loss function, the similarity score includes a first predicted similarity score from image to text and a second predicted similarity score from text to image, the reference score includes a first true similarity score from image to text and a second true similarity score from text to image, and the determining of the loss value between the similarity score and the reference score based on the multimodal supervised contrast loss function includes: determining a first KL divergence based on the first true similarity score and the first predicted similarity score, and determining a second KL divergence based on the second true similarity score and the second predicted similarity score, wherein the KL divergence characterizes the difference between the true similarity score and the predicted similarity score; determining the loss value between the similarity score and the reference score based on the first KL divergence and the second KL divergence, wherein the loss value is the average of the first KL divergence and the second KL divergence.

[0007] Exemplarily, the grain recognition model includes an image encoding module, a text encoding module and a similarity calculation module, and the image encoding module and the text encoding module are both connected to the similarity calculation module; the feature extraction of each grain image and the corresponding text label based on the grain recognition model to obtain image feature data and text feature data includes: extracting the image feature data of each grain image through the image encoding module, and extracting the text feature data of the text label corresponding to each grain image through the text encoding module; the determination of the similarity score between the image feature data and the text feature data based on the grain recognition model includes: calculating the similarity score between the image feature data and the text feature data through the similarity calculation module.

[0008] Exemplarily, the image feature data includes image feature coding data and image feature vector data, and the image coding module includes an image preprocessing submodule, an image feature extraction submodule and an image projection mapping submodule connected in sequence, wherein the image feature data of each grain image is extracted through the image coding module, including: performing a preprocessing operation on the grain image through the image preprocessing submodule, wherein the preprocessing operation includes at least one of image sharpening, image cropping, image flipping, color jittering, and Gaussian blur processing; extracting the features of the preprocessed grain image through the image feature extraction submodule to obtain the image feature coding data; and converting the image feature coding data into the image feature vector through the image projection mapping submodule.

[0009] Exemplarily, the text feature data includes text feature encoding data and text feature vector data, and the text encoding module includes a text preprocessing submodule, a text feature extraction submodule and a text projection mapping submodule connected in sequence, wherein the text feature data of the text label corresponding to each grain image is extracted by the text encoding module, including: performing a preprocessing operation on the text label corresponding to the grain image by the text preprocessing submodule, wherein the preprocessing operation includes at least one of text cleaning, text segmentation, normalization, word embedding, and vectorization processing; extracting the features of the preprocessed text label by the text feature extraction submodule to obtain the text feature encoding data; and converting the text feature encoding data into the text feature vector data by the text projection mapping submodule.

[0010] Exemplarily, the image feature extraction submodule includes an input unit, a first enhancement unit, a first transition unit, a second enhancement unit, a second transition unit, a third enhancement unit, a third transition unit and a fourth enhancement unit connected in sequence, the input unit includes a first convolutional layer, a second convolutional layer and a third convolutional layer connected in sequence, each convolutional layer is connected to an activation function layer; the first enhancement unit includes a local perception subunit and a convolutional feedforward network subunit connected in sequence; the first transition unit includes a fourth convolutional layer and a fifth convolutional layer connected in sequence, each convolutional layer is connected to an activation function layer; the fourth enhancement unit includes a lightweight multi-head attention subunit and a convolutional feedforward network subunit connected in sequence; wherein, the first enhancement unit, the second enhancement unit and the third enhancement unit have the same structure, and the first transition unit, the second transition unit and the third transition unit have the same structure.

[0011] Exemplarily, the first convolution layer includes three branches, wherein the first branch includes a convolution kernel and a normalization layer, the second branch includes a convolution kernel and a normalization layer, and the third branch includes a normalization layer, and the first branch, the second branch and the third branch are all connected to the activation function layer; the third convolution layer includes two branches, wherein the first branch includes a convolution kernel and a normalization layer, and the second branch includes a normalization layer, and the first branch and the second branch are all connected to the activation function layer; the local perception subunit includes a normalization layer and a convolution kernel connected in sequence in a residual manner, and the convolution feedforward network subunit includes a convolution kernel, a normalization layer, a convolution kernel, an activation function layer and a convolution kernel connected in sequence in a residual manner; the lightweight multi-head attention subunit includes three branches, each branch includes a convolution kernel and a linear layer; wherein the second convolution layer and the fourth convolution layer have the same structure as the first convolution layer, and the fifth convolution layer has the same structure as the third convolution layer.

[0012] Exemplarily, the text feature extraction submodule includes multiple conversion encoder units, each of which is connected to a normalization layer, and the conversion encoder unit includes a multi-head attention subunit and a feedforward neural network subunit connected in sequence in a residual manner.

[0013] Exemplarily, the image projection mapping submodule and the text projection mapping submodule include fully connected layers.

[0014] Another embodiment of the present application provides a grain identification method, which includes: obtaining a grain image; inputting the grain image into a trained grain identification model for identification, and obtaining an identification result of the grain image, wherein the identification result includes the grain type and physical properties, and the trained grain identification model is trained according to the above-mentioned grain identification model training method.

[0015] Exemplarily, before inputting the grain image into a trained grain recognition model for recognition and obtaining the recognition result of the grain image, the method also includes: performing a simplification operation on the trained grain recognition model; wherein the simplification operation includes at least one of the following: removing other branch structures except the first branch in the convolution layer, removing the normalization layer and the residual structure in the local perception subunit.

[0016] Exemplarily, the method also includes: using a text encoding module in a trained grain recognition model to perform feature extraction on multiple grain text description words to obtain multiple text feature vector data; wherein the multiple text feature vector data that have been feature extracted in advance are stored in a storage unit corresponding to the trained grain recognition model, so as to calculate the similarity score between the image feature vector data and the text feature vector data, so as to obtain the recognition result of the grain image based on the similarity score.

[0017] Another embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method of any of the above embodiments when executing the computer program.

[0018] Another embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the method of any of the above embodiments are implemented.

[0019] Another embodiment of the present application provides a computer program product, which includes instructions. When the instructions are executed by a processor of a computer device, the computer device is enabled to perform the steps of the method of any of the above embodiments.

[0020] In the above implementation, a plurality of grain images and text labels corresponding to each grain image are obtained for model training; a grain recognition model to be trained is constructed, and feature extraction is performed on each grain image and the corresponding text label based on the grain recognition model to obtain image feature data and text feature data; a similarity score between the image feature data and the text feature data is determined based on the grain recognition model; a multimodal supervised contrast loss function is constructed, and a loss value between the similarity score and the reference score is determined based on the multimodal supervised contrast loss function, and the model parameters of the grain recognition model are adjusted based on the loss value, so as to iteratively train the model and obtain a trained grain recognition model. The grain recognition model of the present invention can be widely used in various types of grain detection tasks to improve detection efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A flowchart of a method for training a grain recognition model provided in an embodiment of the present application;

[0022] Figure 2 A schematic diagram of the structure of a grain recognition model provided in an embodiment of the present application;

[0023] Figure 3 A flowchart for extracting image feature data of each grain image provided by an embodiment of the present application;

[0024] Figure 4A schematic diagram of the structure of the image feature extraction submodule provided in the embodiment of the present application;

[0025] Figure 5 A schematic diagram of the structure of an input unit provided in an embodiment of the present application;

[0026] Figure 6 A schematic diagram of the structure of a first enhancement unit provided in an embodiment of the present application;

[0027] Figure 7 A schematic diagram of the structure of a convolutional feedforward network subunit provided in an embodiment of the present application;

[0028] Figure 8 A schematic diagram of the structure of a fourth enhancement unit provided in an embodiment of the present application;

[0029] Fig. 9 A schematic diagram of the structure of a first transition unit provided in an embodiment of the present application;

[0030] Fig.10 A flowchart of extracting text feature data corresponding to text labels of grain images provided in an embodiment of the present application;

[0031] Fig.11 A flowchart for determining a loss value between a similarity score and a reference score provided for an embodiment of the present application;

[0032] Fig.12 A flowchart of a grain identification method provided for an embodiment of the present application;

[0033] Fig.13 A schematic diagram of offline extraction of text description word vector data provided by an embodiment of the present application;

[0034] Fig.14 A schematic diagram of a process for detecting imperfect grains of wheat grains provided in an embodiment of the present application;

[0035] Fig.15 A flow chart of a multimodal cross-grain imperfect grain detection model provided for an embodiment of the present application;

[0036] Fig.16 A block diagram of an electronic device provided for an embodiment of the present application. DETAILED DESCRIPTION

[0037] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0038] This application proposes a training method for a grain recognition model. The trained grain recognition model is a multimodal model that utilizes text information and image information. It can also provide a cross-grain imperfect grain detection function covering a variety of grain types, and can be widely used in a variety of grain detection tasks. Such a model can not only improve detection efficiency and reduce the impact of imperfect grains on product quality, but also help to accurately assess grain quality and food security.

[0039] Figure 1 This is a flowchart of a method for training a grain recognition model according to an embodiment of the present application.

[0040] like Figure 1 As shown, the training method of the grain recognition model includes:

[0041] S101, obtaining multiple grain images for model training and a text label corresponding to each grain image.

[0042] S102, constructing a grain recognition model to be trained, and performing feature extraction on each grain image and corresponding text label based on the grain recognition model to obtain image feature data and text feature data.

[0043] S103, determining a similarity score between the image feature data and the text feature data based on the grain recognition model.

[0044] S104, constructing a multimodal supervised contrast loss function, and determining a loss value between a similarity score and a reference score based on the multimodal supervised contrast loss function, and adjusting model parameters of the grain recognition model based on the loss value, so as to iteratively train the model to obtain a trained grain recognition model.

[0045] Exemplarily, multiple grain images for training and text labels corresponding to each grain image are obtained. For example, multiple types of grain text-image pair data sets for training can be prepared, and each grain image has a corresponding text label, such as "corn spots", "sorghum sprouts", etc. The grain recognition model to be trained is constructed with a convolutional neural network architecture. The grain recognition model can perform feature extraction on each grain image and the corresponding text label respectively, obtain image feature data and text feature data, and calculate the similarity score between the image feature data and the text feature data. Then, a multimodal supervised contrast loss function is constructed, and the loss value between the similarity score and the reference score is calculated according to the multimodal supervised contrast loss function. The model parameters of the grain recognition model are adjusted based on the loss value so that the model can be iteratively trained to obtain a trained grain recognition model.

[0046] It can be understood that the similarity score between the image feature data and the text feature data calculated by the grain recognition model is the similarity score predicted by the model, and the reference score is the real similarity score. The model parameters of the grain recognition model are adjusted according to the loss value between the similarity score and the reference score. The model parameters of the grain recognition model are adjusted with the smaller loss value as the goal. The smaller the loss value, the closer the similarity score predicted by the model is to the real similarity score, that is, the better the prediction ability of the grain recognition model. After adjusting the parameters of the grain recognition model, iterative training continues to obtain a trained grain recognition model. For example, the number of iterations can be set as the termination condition, and the training is terminated when the number of iterations reaches the preset number of iterations. Of course, the termination condition can also be other conditions such as model accuracy, such as when the model accuracy reaches the preset accuracy, the training is terminated.

[0047] The grain recognition model of the present application covers the imperfect grain detection function across grains of various grain types, and can be widely used in various grain detection tasks to improve detection efficiency.

[0048] As an example, a text-image pair dataset of various types of grains for training can be created first, such as wheat, corn, sorghum, paddy, brown rice, and rice. Each image has a corresponding text description word, indicating its related image appearance attributes. The text description word contains the type of grain and appearance characteristics corresponding to the image. For example, if a picture is a picture of wheat grains, and the appearance shows the characteristics of disease spots, the text description word for this picture is "wheat disease spots". Similarly, the descriptors for other grain pictures can be "corn disease spots", "sorghum sprouts", "rice grains are shriveled", "brown rice is not ripe", "rice worm-eaten", etc. Detailed label description words are shown in Table 1 below.

[0049] Table 1

[0050]

[0051] As an example, Figure 2 As shown, the grain recognition model includes an image encoding module, a text encoding module and a similarity calculation module, and the image encoding module and the text encoding module are both connected to the similarity calculation module; based on the grain recognition model, feature extraction is performed on each grain image and the corresponding text label to obtain image feature data and text feature data, including: extracting the image feature data of each grain image through the image encoding module, and extracting the text feature data of the text label corresponding to each grain image through the text encoding module; determining the similarity score between the image feature data and the text feature data based on the grain recognition model, including: calculating the similarity score between the image feature data and the text feature data through the similarity calculation module.

[0052] Exemplarily, the grain recognition model in the present application includes two independent unimodal image encoding modules and a text encoding module, as well as a similarity calculation module, wherein the image encoding module further includes an image preprocessing submodule, an image feature extraction submodule, and an image projection mapping submodule connected in sequence. The text encoding module includes a text preprocessing submodule, a text feature extraction submodule, and a text projection mapping submodule connected in sequence. The image encoding module extracts the image feature data of each grain image, and the text encoding module extracts the text feature data of the text label corresponding to each grain image. Both the image encoding module and the text encoding module are connected to a similarity calculation module, which is used to calculate the similarity score between the image feature data and the text feature data.

[0053] The image encoding module is described in detail below.

[0054] As an example, Figure 3 As shown, the image feature data includes image feature coding data and image feature vector data, and the image coding module includes an image preprocessing submodule, an image feature extraction submodule and an image projection mapping submodule connected in sequence, wherein the image feature data of each grain image is extracted by the image coding module, including:

[0055] S301, performing a preprocessing operation on the grain image through an image preprocessing submodule, wherein the preprocessing operation includes at least one of image sharpening, image cropping, image flipping, color dithering, and Gaussian blur processing.

[0056] S302, extracting features of the preprocessed grain image through an image feature extraction submodule to obtain image feature encoding data.

[0057] S303, converting the image feature encoding data into an image feature vector through an image projection mapping submodule.

[0058] Exemplarily, the image encoding module includes an image preprocessing submodule, an image feature extraction submodule and an image projection mapping submodule connected in sequence. The image preprocessing submodule performs preprocessing operations on the input image, and includes image sharpening, image cropping, image horizontal and vertical flipping, color jitter and Gaussian blur and other preprocessing operations in the training stage. In the inference stage, in order to improve the efficiency of the model, the image preprocessing submodule may only include the function of adjusting the image size. Of course, the inference stage may also retain the preprocessing operations in the training stage. Afterwards, the image feature extraction submodule extracts the features of the preprocessed grain image to obtain image feature encoding data, and the image projection mapping submodule converts the image feature encoding data into an image feature vector. The image feature extraction submodule is a convolutional neural network, and the image feature extraction submodule can be a separate learnable single hidden layer neural network.

[0059] As an example, Figure 4 As shown, the image feature extraction submodule includes an input unit, a first enhancement unit, a first transition unit, a second enhancement unit, a second transition unit, a third enhancement unit, a third transition unit and a fourth enhancement unit connected in sequence, the input unit includes a first convolutional layer, a second convolutional layer and a third convolutional layer connected in sequence, each convolutional layer is connected to an activation function layer; the first enhancement unit includes a local perception subunit and a convolutional feedforward network subunit connected in sequence; the first transition unit includes a fourth convolutional layer and a fifth convolutional layer connected in sequence, each convolutional layer is connected to an activation function layer; the fourth enhancement unit includes a lightweight multi-head attention subunit and a convolutional feedforward network subunit connected in sequence; wherein the first enhancement unit, the second enhancement unit and the third enhancement unit have the same structure, and the first transition unit, the second transition unit and the third transition unit have the same structure.

[0060] For example, Figure 4 As shown, the image feature extraction submodule includes an input unit, a first enhancement unit, a first transition unit, a second enhancement unit, a second transition unit, a third enhancement unit, a third transition unit and a fourth enhancement unit connected in sequence. The input unit can also be called a stem unit, and the first enhancement unit, the second enhancement unit, the third enhancement unit and the fourth enhancement unit are composed of multiple convolutional neural networks, which can also be called stage1, stage2, stage3, and stage4. Among them, the structures of the first three enhancement units are the same. The transition unit between the enhancement units is also composed of multiple convolutional neural networks, which can be called a Patch Embedding unit, and the structures of the first transition unit, the second transition unit and the third transition unit are the same.

[0061] Exemplarily, the image feature extraction module of the present application uses a convolutional neural network to process grain images to obtain high-dimensional features of the image. For example, the RepCMT architecture model can be used for image feature extraction. The RepCMT architecture model is a hybrid visual network that gives full play to the advantages of CNNs and Transformers. The uniqueness of this model is that it uses linear training time over-parameterization technology to over-parameterize the layers of the k×k deep convolutional layer during training to improve the accuracy of the model during training. The RepCMT model architecture consists of an input unit Stem, three transition units PatternBedding, and four enhancement units stage.

[0062] The input unit Stem is described in detail below.

[0063] As an example, Figure 5As shown, the input unit includes a first convolutional layer, a second convolutional layer, and a third convolutional layer connected in sequence, and each convolutional layer is connected to an activation function layer. The first convolutional layer includes three branches, wherein the first branch includes a convolution kernel and a normalization layer, the second branch includes a convolution kernel and a normalization layer, the third branch includes a normalization layer, and the first branch, the second branch, and the third branch are all connected to the activation function layer. The structure of the second convolutional layer is the same as that of the first convolutional layer, and both include three branches. The third convolutional layer includes two branches, wherein the first branch includes a convolution kernel and a normalization layer, the second branch includes a normalization layer, and the first branch and the second branch are both connected to the activation function layer.

[0064] Exemplarily, the first convolution layer may adopt a 3×3 convolution layer, the second convolution layer may adopt a 3×3 depth separation convolution layer, and the third convolution layer may adopt a 1×1 point-by-point convolution layer, and each convolution layer is followed by a BN normalization layer and a GELU activation function layer. During the training stage, the linear training time over-parameterization technique is used to increase the width of the convolution layer. For example, a 1×1 convolution layer branch and a BN normalization layer branch are added to the 3×3 convolution layer, a 1×1 convolution layer branch and a BN normalization layer branch are added to the 3×3 depth separation convolution, and a BN normalization layer branch is added to the 1×1 point-by-point convolution layer.

[0065] This application uses large kernel convolution to replace the self-attention mechanism in the first three enhancement unit stages of the model to reduce the complexity of the model. The first enhancement unit, the second enhancement unit, and the third enhancement unit have the same structure, and the first enhancement unit is used as an example for detailed description below.

[0066] As an example, Figure 6 As shown, the first enhancement unit includes a local perception subunit and a convolutional feedforward network subunit connected in sequence. The local perception subunit includes a normalization layer and a convolution kernel connected in sequence in a residual manner. Figure 7 As shown, the convolutional feedforward network subunit includes a convolution kernel, a normalization layer, a convolution kernel, an activation function layer and a convolution kernel connected in sequence in a residual manner.

[0067] Exemplarily, the first three enhancement units stage1-3 all use the same basic components, consisting of a local perception subunit LPU and a convolutional feedforward network subunit ConvFFN. The local perception subunit is composed of a normalization layer, a 3×3 depth separation convolution, and a residual connection method is used. The convolutional feedforward network subunit is composed of a 7×7 depth separation convolution plus a 1×1 point-by-point convolution layer, a BN normalization layer, an activation function layer, and a 1×1 convolution, and a residual connection method is used. Using a 7×7 large-core convolution can effectively increase the size of the receptive field while controlling the amount of calculation. Replacing the self-attention mechanism is an efficient and time-saving method. At the same time, large-core convolution also helps to improve the robustness of the model.

[0068] As an example, Figure 8 As shown, the fourth enhancement unit includes a lightweight multi-head attention subunit and a convolutional feedforward network subunit connected in sequence. The lightweight multi-head attention subunit includes three branches, each of which includes a convolution kernel and a linear layer.

[0069] Exemplarily, in order to increase the performance and robustness of the model, the fourth enhancement unit stage4 does not use the structural reparameterization technology, but is composed of a lightweight multi-head attention subunit LMHSA and a convolutional feedforward network subunit ConvFFN, where the LMHSA module is a lightweight self-attention mechanism. Specifically, for the input image feature map , use a depth-separable convolution with a step size of s and a convolution kernel size of k×k and a linear layer to linearly transform the input image feature map to obtain the K (Key) vector K , V (value) vector and Q(query) vector , where n= In actual applications, the step size of the depthwise separable convolution used to obtain the K and V vectors can be 2, and the convolution kernel size can be 2 2. The step size of the depthwise separable convolution used to obtain the Q vector can be 1, and the convolution kernel size can be 1 1. A relative position deviation B is added to each self-attention module, and the corresponding LMHSA lightweight attention calculation formula is as follows:

[0070]

[0071] in, is randomly initialized and learnable, is the dimension of the K vector.

[0072] The first transition unit, the second transition unit, and the third transition unit have the same structure, and the first transition unit is taken as an example for detailed description below.

[0073] As an example, Fig. 9 As shown, the first transition unit includes a fourth convolution layer and a fifth convolution layer connected in sequence, each convolution layer is connected to an activation function layer, the fourth convolution layer has the same structure as the first convolution layer, and the fourth convolution layer includes three branches, wherein the first branch includes a convolution kernel and a normalization layer, the second branch includes a convolution kernel and a normalization layer, the third branch includes a normalization layer, and the first branch, the second branch and the third branch are all connected to the activation function layer. The fifth convolution layer has the same structure as the third convolution layer, and the fifth convolution layer includes two branches, wherein the first branch includes a convolution kernel and a normalization layer, the second branch includes a normalization layer, and the first branch and the second branch are both connected to the activation function layer.

[0074] Exemplarily, the first transition unit may be composed of a 7×7 large convolution kernel depth separation convolution layer and a 1×1 point-by-point convolution layer. Each convolution layer is followed by a normalization layer and an activation function layer. During the training phase, the linear training time over-parameterization technique is used to increase the width of the convolution layer. A 3×3 depth separation convolution layer branch is added to the 7×7 large convolution kernel depth separation convolution layer, and a normalization layer branch is added to the 1×1 point-by-point convolution layer.

[0075] The image feature extraction submodule is used to extract features from the grain image to obtain image feature coding data, and the image projection mapping submodule converts the image feature coding data into image feature vector data.

[0076] As an example, the image projection mapping submodule includes a fully connected layer.

[0077] Exemplarily, the image projection mapping submodule maps the features extracted from the image features into a fixed d1-dimensional vector space (where d1 is related to the number of images input in a batch). The image projection mapping submodule is a separate learnable single hidden layer neural network that can use an fc fully connected layer.

[0078] The text encoding module is described in detail below.

[0079] As an example, Fig.10 As shown, the text feature data includes text feature encoding data and text feature vector data, and the text encoding module includes a text preprocessing submodule, a text feature extraction submodule and a text projection mapping submodule connected in sequence, wherein the text feature data of the text label corresponding to each grain image is extracted by the text encoding module, including:

[0080] S1001, performing a preprocessing operation on the text label corresponding to the grain image through a text preprocessing submodule, wherein the preprocessing operation includes at least one of text cleaning, text segmentation, normalization, word embedding, and vectorization processing.

[0081] S1002, extracting features of the preprocessed text label through a text feature extraction submodule to obtain text feature encoding data.

[0082] S1003, converting the text feature encoding data into text feature vector data through a text projection mapping submodule.

[0083] Exemplarily, the text encoding module includes a text preprocessing submodule, a text feature extraction submodule and a text projection mapping submodule connected in sequence. The text preprocessing submodule performs a series of preprocessing on the text descriptor data of the image, and the preprocessing operation includes at least one of text cleaning, text segmentation, normalization, word embedding, and vectorization. Afterwards, the text feature extraction submodule extracts the features of the preprocessed text label to obtain text feature encoding data, and the text projection mapping submodule converts the text feature encoding data into text feature vector data. The text feature extraction submodule is a convolutional neural network, and the text feature extraction submodule can be a separate learnable single hidden layer neural network.

[0084] As an example, the text feature extraction submodule includes multiple conversion encoder units, each of which is connected to a normalization layer, and the conversion encoder unit includes a multi-head attention subunit and a feedforward neural network subunit connected in sequence in a residual manner.

[0085] Exemplarily, the text feature extraction submodule adopts the BERT model, which is composed of multiple stacked Transformer encoders, each of which consists of multiple layers. The Transformer encoder is an architecture based on self-attention mechanism and position encoding, and is constructed through components such as multi-head attention, feedforward neural network, residual connection and layer normalization. The multi-head attention mechanism is the core part of the BERT model, which can efficiently capture long-distance dependencies in the input sequence. A feedforward neural network is connected after the attention layer, which helps to capture the nonlinear relationship between texts. There are residual connections between each sub-unit, which helps to avoid gradients that are too small or gradient explosions, and improves training efficiency. After each sub-module, a normalization layer is added to normalize the output of each sub-module, which enhances the stability of the model and the training effect.

[0086] As an example, the text projection mapping submodule includes a fully connected layer.

[0087] Exemplarily, the text projection mapping submodule maps the features extracted from the text features into a fixed d2-dimensional vector space (where d2 is related to the number of text labels input in a batch). The text projection mapping submodule is a separate learnable single hidden layer neural network that can use an FC fully connected layer.

[0088] Based on the above-mentioned grain recognition model structure, a grain recognition model to be trained is constructed, image feature data of each grain image is extracted based on the image encoding module, text feature data of the text label corresponding to each grain image is extracted based on the text encoding module, and the similarity calculation module calculates the similarity score between the image feature data and the text feature data.

[0089] The following is a detailed description of the multimodal supervised contrast loss function and the loss value between the similarity score and the reference score determined based on the multimodal supervised contrast loss function.

[0090] At present, the multimodal learning method based on text-image contrast is to jointly train the image encoder and the text encoder to predict the correct pairing of a batch of (image-text pairs) training examples. The cross-entropy loss function is used to optimize the text-image contrast model, and each image in a batch of data corresponds to a different text label by default. In the actual training scenario, since the number of grain images in a batch is much larger than the fixed text labels, and some single grain images contain multiple imperfect grain features (wheat grains also have the appearance features of imperfect grains of insect damage and mold), corresponding to multiple text labels, multiple text labels will appear in a batch of data and match the same image, and a text label may also match multiple images. By default, each image in a batch of data corresponds to a different text label. It is inappropriate to use the cross-entropy loss function to optimize the text-image contrast multimodal model.

[0091] Based on this, this application defines a new image-text contrast loss function, and uses the KL divergence of the text and image feature similarity matrix and the label similarity matrix to optimize the multimodal cross-grain imperfect grain detection model, bringing the matching text and image features closer to each other.

[0092] As an example, Fig.11 As shown, the loss value between the similarity score and the reference score is determined based on the multimodal supervised contrast loss function, including:

[0093] S1101, determining a first KL divergence based on a first true similarity score and a first predicted similarity score, and determining a second KL divergence based on a second true similarity score and a second predicted similarity score, wherein the KL divergence represents a difference between the true similarity score and the predicted similarity score.

[0094] S1102, determining a loss value between the similarity score and the reference score based on the first KL divergence and the second KL divergence, wherein the loss value is an average value of the first KL divergence and the second KL divergence.

[0095] Exemplarily, the multimodal supervised contrast loss function includes a KL divergence multimodal supervised contrast loss function, the similarity score includes a first predicted similarity score from image to text and a second predicted similarity score from text to image, and the reference score includes a first true similarity score from image to text and a second true similarity score from text to image.

[0096] For example, in the training phase, a batch of grain images x are given, and the text label descriptors y corresponding to these grain images are given to train the grain recognition model. and text encoding modules To learn a multimodal embedding space to maximize the cosine similarity between each image in this batch and its corresponding text label description, while minimizing the cosine similarity of mismatched image-text pairs. Image Encoding Module Used to extract features of input grain images, text encoding module It is used to extract the features of the input text label. Then, in order to bring the paired images and the corresponding text label descriptors close to each other, the symmetric similarity between the two modalities is defined using the cosine distance in the similarity calculation module:

[0097]

[0098] in, are the encoded features of the grain image x and the text label descriptor y, respectively, then the image-to-text and text-to-image similarity scores can be calculated as:

[0099]

[0100]

[0101] in, is the first predicted image-to-text similarity score for the i-th grain image, is the second predicted text-to-image similarity score for the jth text label, is a learnable temperature parameter, and N is the number of images and text labels in a batch.

[0102] Define the KL divergence multimodal supervised contrast loss function L:

[0103]

[0104] in, is the first truth similarity score from image to text, is the second true similarity score from text to image, is the first predicted similarity score from image to text, is the second predicted similarity score from text to image, is the sigmoid activation function. In which, the probability of a negative pair is 0 and the probability of a positive pair is 1 (e.g. Figure 2 As shown in Figure 2.1, it can be understood that the probability is 1 when the image and text match, and the probability is 0 when the image and text do not match. It can be understood that p is the similarity score predicted by the above formula (model), because the images and corresponding text labels in the dataset are known, and q is the true value representing the similarity score.

[0105] A first KL divergence is determined based on the first true similarity score and the first predicted similarity score. The first KL divergence is: Based on the second true similarity score and the second predicted similarity score, a second KL divergence is determined, and the second KL divergence is A loss value between the similarity score and the reference score is determined based on the first KL divergence and the second KL divergence. As shown in the above formula, the loss value L is an average value of the first KL divergence and the second KL divergence.

[0106] Among them, KL divergence is a method to measure the difference between two probability distributions. It measures the information loss generated when describing two probability distributions of the same random variable in the same time and space. Specifically, for two probability distributions P and Q, KL divergence is defined as the relative entropy of P relative to Q, and the formula is:

[0107]

[0108] Where x represents the value of a random variable. and They represent the probabilities of P and Q corresponding to x, respectively. The larger the KL divergence value, the more dissimilar the two distributions are, and the smaller the value, the more similar the two distributions are. The grain recognition model is trained with the smaller KL value as the goal. The smaller the KL value, the more accurate the model's prediction ability.

[0109] As an example, multiple grain images and text labels corresponding to each grain image obtained for model training are input into the grain recognition model to train the grain recognition model, update the network weights, and optimize the network model. Use KL divergence multimodal supervised contrast loss function To optimize the parameters of the text encoder and image encoder of the grain recognition model, until the number of iterations reaches the preset threshold, the final training model is obtained. The AdamW optimizer is used in the training process, and the weight decay parameter is , the batch size is 256, the image resolution of the image encoder is 256×256, and the basic learning rate for modules with learnable parameters is , and gradually decreases as the number of training iterations increases. The number of training iterations can be set to 50 times. When the number of iterations is reached, the network training is terminated to obtain a multimodal cross-grain deep learning model, that is, a trained grain recognition model.

[0110] This application also proposes a grain identification method.

[0111] like Fig.12 As shown, the grain identification method includes:

[0112] S1201, obtaining a grain image.

[0113] S1202, inputting the grain image into a trained grain recognition model for recognition, and obtaining a recognition result of the grain image, wherein the recognition result includes the grain type and physical properties, and the trained grain recognition model is trained according to the training method of the grain recognition model of the above embodiment.

[0114] As an example, the grain identification method also includes: using the text encoding module in the trained grain identification model to extract features of multiple grain text description words to obtain multiple text feature vector data; wherein the multiple text feature vector data that have been feature extracted in advance are stored in the storage unit corresponding to the trained grain identification model, so as to calculate the similarity score between the image feature vector data and the text feature vector data, so as to obtain the recognition result of the grain image based on the similarity score.

[0115] For example, in order to improve the inference speed of the grain recognition model in detecting grains, for the vector of the text description word, the text encoding module in the trained grain recognition model is first used, such as Fig.13 As shown, all the cereal category text description words in Table 1 in the training data are encoded to generate corresponding text feature vector data, and the text descriptor labels and their corresponding text feature vector data are stored in a storage unit, which can be a JSON file.

[0116] As an example, in the process of detecting the category of cereal grains, we first determine the type of cereal to be detected and the category of cereal grains to be identified, load the vector representation of the required text prompt words from the saved JSON file for comparison with the image vector, and then use the image encoding module in the trained cereal recognition model to convert the image into an image vector representation, and use cosine similarity to calculate the similarity between the image vector and the text prompt word vector.

[0117] For example, Fig.14 As shown in the figure, the imperfect grains of wheat grains are detected, and the categories of wheat grains to be detected are normal, diseased, sprouted, moldy, worm-eaten, damaged, black embryo and mildewed. Then, the vectors of these text prompt words saved in the JSON file are read based on the description words of the categories of wheat grains as the key values, as shown in the figure. Fig.14 T 1 -T 9 , the input grain image is converted into an image vector representation through the image encoder model, such as Fig.14 Shown I 1 , use the similarity calculation module to calculate the vector of each text prompt word and the image vector I 1 The category with the largest similarity is taken as the recognition result of the wheat grain particles. For example, the recognition result of the wheat grain particles is “wheat worm-eaten”.

[0118] For example, the imperfect grains of sorghum grains are detected and the detected category is sorghum, and the detected categories are normal, diseased, sprouted, moldy, worm-eaten, damaged, heat-damaged and mildewed. The vectors of these text prompt words saved in the JSON file are read according to the category description words of sorghum grains as key values, and the input grain grain image is converted into an image vector representation through the image encoder model. The similarity between the image vector and the text prompt word vector is calculated using the similarity calculation module, and the category with the largest similarity is taken as the recognition result of the sorghum grain grain.

[0119] Similarly, this method can also be used to detect imperfect grains of other grains such as corn, paddy, brown rice, rice, soybeans, etc.

[0120] As an example, before inputting the grain image into a trained grain recognition model for recognition and obtaining the recognition result of the grain image, the method also includes: performing a simplification operation on the trained grain recognition model; wherein the simplification operation includes at least one of the following: removing other branch structures in the convolution layer except the first branch, removing the normalization layer and the residual structure in the local perception subunit.

[0121] Exemplarily, in order to improve the inference speed of the grain recognition model in detecting grains, the trained grain recognition model is simplified, and the simplification operation includes removing other branch structures except the first branch in the convolutional layer, and removing at least one of the normalization layer and residual structure in the local perception subunit.

[0122] For example, for the image coding module RepCMT model, the structural reparameterization technology is used to delete the residual connection and the over-parameterized extra branches in the model to reduce the memory access cost, and TensorRT is used for model conversion, making full use of TensorRT's GPU hardware acceleration to significantly improve the image encoder inference speed to achieve more efficient image processing tasks. For example, for the input unit, the structural reparameterization technology is used in the inference stage to remove the branch structure except the first branch in the convolution layer, which consists of a 3×3 convolution layer, a 3×3 depth separation convolution layer, a 1×1 convolution layer, and a GELU activation function layer.

[0123] Fig.15 This is a flow chart of a multimodal cross-grain imperfect grain detection model according to an embodiment of the present application.

[0124] like Fig.15 As shown in the figure, firstly, a multi-grain text-image pair dataset is constructed, and each image is accompanied by a description word, including the grain type and appearance features, such as "wheat spots". A multimodal cross-grain imperfect grain detection model is created, which includes image and text preprocessing modules, image encoder module, text encoder module, and similarity calculation module. The KL divergence multimodal supervision loss function is designed to learn the multimodal embedding space by maximizing the similarity of true pairs and minimizing the similarity of false matching pairs. Input the training data, use the AdamW optimizer and the KL divergence multimodal supervision loss function to train the multimodal cross-grain imperfect grain detection model, and update the weights to reach the preset number of iterations. Use the trained text encoder module to encode the grain category text description words, and save the text vector in JSON data exchange format. The inference speed of the image encoder is improved by structural reparameterization and TensorRT optimization. Determine the detected grain type and imperfect grain category, load the text prompt word vector of the detected grain category saved in the JSON file, use the image encoder to convert the detected grain image into an image vector, and select the most similar category through the similarity calculation module to realize the recognition of grain grains.

[0125] The multimodal cross-grain imperfect grain detection model of this application has great potential significance for grain quality inspection. With the diversification of market demand for agricultural products and the improvement of quality requirements, a detection model that can cross different types of grains will become a very valuable tool. The grain detection model of this application can not only improve detection efficiency and reduce the impact of imperfect grains on product quality, but also help to accurately evaluate grain quality and food security.

[0126] The application also proposes a computer-readable storage medium.

[0127] In this embodiment, a computer program is stored on a computer-readable storage medium, and when the computer program is executed by a processor, the steps of the training method of the grain identification model and the grain identification method are implemented.

[0128] Fig.16 A block diagram of an electronic device provided for an embodiment of the present application.

[0129] An embodiment of the present application provides an electronic device including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the training method of the grain recognition model and the grain recognition method are implemented.

[0130] like Fig.16 As shown, for ease of understanding, the embodiment of the present application shows a specific electronic device.

[0131] Electronic devices are intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0132] like Fig.16 As shown, the device includes a computing unit 1601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1602 or a computer program loaded from a storage unit 1608 into a random access memory (RAM) 1603. In RAM 1603, various programs and data required for the operation of the electronic device can also be stored. The computing unit 1601, ROM 1602, and RAM 1603 are connected to each other via a bus 1604. An input / output (I / O) interface 1605 is also connected to the bus 1604.

[0133] Multiple components in the electronic device are connected to the I / O interface 1605, including: an input unit 1606, such as a keyboard, a mouse, etc.; an output unit 1607, such as various types of displays, speakers, etc.; a storage unit 1608, such as a disk, an optical disk, etc.; and a communication unit 1609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1609 allows the electronic device to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0134] The computing unit 1601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1601 executes the various methods described above, such as the training method of the grain recognition model and the grain recognition method. For example, in some embodiments, the training method of the grain recognition model and the grain recognition method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 1608. In some embodiments, part or all of the computer program may be loaded and / or installed on an electronic device via the ROM 1602 and / or the communication unit 1609. When the computer program is loaded into the RAM 1603 and executed by the computing unit 1601, the training method of the grain recognition model and the grain recognition method described above may be executed. Alternatively, in other embodiments, the computing unit 1601 may be configured to execute the grain identification model training method and the grain identification method in any other appropriate manner (eg, by means of firmware).

[0135] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this application, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection portion with one or more wirings (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and editable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or otherwise processing in a suitable manner if necessary, and then stored in a computer memory.

[0136] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0137] In the description of the present application, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In the present application, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0138] In the description of the present application, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", "circumferential" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.

[0139] In addition, the terms "first", "second", etc. used in the embodiments of the present application are only used for descriptive purposes and should not be understood as indicating or implying relative importance, or implicitly indicating the number of technical features indicated in the present embodiment. Therefore, the features defined by the terms "first", "second", etc. in the embodiments of the present application can explicitly or implicitly indicate that at least one of the features is included in the embodiment. In the description of the present application, the word "multiple" means at least two or two or more, such as two, three, four, etc., unless otherwise clearly and specifically defined in the embodiments.

[0140] In this application, unless otherwise clearly specified or limited in the embodiments, the terms "installed", "connected", "connected" and "fixed" etc. appearing in the embodiments should be understood in a broad sense. For example, the connection can be a fixed connection, a detachable connection, or an integrated connection. It can be understood that it can also be a mechanical connection, an electrical connection, etc.; of course, it can also be a direct connection, or an indirect connection through an intermediate medium, or it can be the internal connection of two elements, or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to the specific implementation situation.

[0141] In the present application, unless otherwise clearly specified and limited, a first feature being “above” or “below” a second feature may mean that the first and second features are in direct contact, or the first and second features are in indirect contact through an intermediate medium. Moreover, a first feature being “above”, “above”, and “above” a second feature may mean that the first feature is directly above or obliquely above the second feature, or simply means that the first feature is higher in level than the second feature. A first feature being “below”, “below”, and “below” a second feature may mean that the first feature is directly below or obliquely below the second feature, or simply means that the first feature is lower in level than the second feature.

[0142] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A method for training a grain recognition model, characterized in that: The method comprises: Obtain multiple grain images and text labels corresponding to each grain image for model training; Constructing a grain recognition model to be trained, and performing feature extraction on each grain image and corresponding text label based on the grain recognition model to obtain image feature data and text feature data; determining a similarity score between the image feature data and the text feature data based on the grain recognition model; Constructing a multimodal supervised contrast loss function, and determining a loss value between the similarity score and the reference score based on the multimodal supervised contrast loss function, and adjusting a model parameter of the grain recognition model based on the loss value, so as to iteratively train the model to obtain a trained grain recognition model; wherein the reference score indicates a true similarity score between the image feature data and the text feature data; The multimodal supervised contrast loss function includes a KL divergence multimodal supervised contrast loss function, the similarity score includes a first predicted similarity score from image to text and a second predicted similarity score from text to image, the reference score includes a first true similarity score from image to text and a second true similarity score from text to image, and the loss value between the similarity score and the reference score is determined based on the multimodal supervised contrast loss function, including: determining a first KL divergence based on the first true similarity score and the first predicted similarity score, and determining a second KL divergence based on the second true similarity score and the second predicted similarity score, wherein the KL divergence represents a difference between the true similarity score and the predicted similarity score; A loss value between the similarity score and a reference score is determined based on the first KL divergence and the second KL divergence, wherein the loss value is an average value of the first KL divergence and the second KL divergence.

2. The method for training a grain recognition model according to claim 1, characterized in that: The grain recognition model includes an image encoding module, a text encoding module and a similarity calculation module, and the image encoding module and the text encoding module are both connected to the similarity calculation module; The step of extracting features from each grain image and the corresponding text label based on the grain recognition model to obtain image feature data and text feature data includes: extracting the image feature data of each grain image through the image encoding module, and extracting the text feature data of the text label corresponding to each grain image through the text encoding module; The determining the similarity score between the image feature data and the text feature data based on the grain recognition model includes: calculating the similarity score between the image feature data and the text feature data by the similarity calculation module.

3. The method for training a grain recognition model according to claim 2, characterized in that: The image feature data includes image feature coding data and image feature vector data, and the image coding module includes an image preprocessing submodule, an image feature extraction submodule and an image projection mapping submodule connected in sequence, wherein the image feature data of each grain image is extracted by the image coding module, including: Performing a preprocessing operation on the grain image through the image preprocessing submodule, wherein the preprocessing operation includes at least one of image sharpening, image cropping, image flipping, color jittering, and Gaussian blur processing; Extracting the features of the preprocessed grain image through the image feature extraction submodule to obtain the image feature encoding data; The image feature encoding data is converted into the image feature vector data through the image projection mapping submodule.

4. The method for training a grain recognition model according to claim 3, characterized in that: The text feature data includes text feature encoding data and text feature vector data, and the text encoding module includes a text preprocessing submodule, a text feature extraction submodule and a text projection mapping submodule connected in sequence, wherein the text feature data of the text label corresponding to each grain image is extracted by the text encoding module, including: Performing a preprocessing operation on the text label corresponding to the grain image through the text preprocessing submodule, wherein the preprocessing operation includes at least one of text cleaning, text segmentation, normalization, word embedding, and vectorization processing; Extracting the features of the preprocessed text labels through the text feature extraction submodule to obtain the text feature encoding data; The text feature encoding data is converted into the text feature vector data through the text projection mapping submodule.

5. The method for training a grain recognition model according to claim 3, characterized in that: The image feature extraction submodule includes an input unit, a first enhancement unit, a first transition unit, a second enhancement unit, a second transition unit, a third enhancement unit, a third transition unit and a fourth enhancement unit connected in sequence, The input unit includes a first convolutional layer, a second convolutional layer and a third convolutional layer connected in sequence, and each convolutional layer is connected to an activation function layer; The first enhancement unit includes a local perception subunit and a convolutional feedforward network subunit connected in sequence; The first transition unit includes a fourth convolutional layer and a fifth convolutional layer connected in sequence, and each convolutional layer is connected to an activation function layer; The fourth enhancement unit includes a lightweight multi-head attention subunit and a convolutional feedforward network subunit connected in sequence; The first reinforcing unit, the second reinforcing unit and the third reinforcing unit have the same structure, and the first transition unit, the second transition unit and the third transition unit have the same structure.

6. The method for training a grain recognition model according to claim 5, characterized in that: The first convolution layer includes three branches, wherein the first branch includes a convolution kernel and a normalization layer, the second branch includes a convolution kernel and a normalization layer, the third branch includes a normalization layer, and the first branch, the second branch and the third branch are all connected to the activation function layer; The third convolutional layer includes two branches, wherein the first branch includes a convolution kernel and a normalization layer, the second branch includes a normalization layer, and the first branch and the second branch are both connected to the activation function layer; The local perception subunit includes a normalization layer and a convolution kernel connected in sequence in a residual manner, and the convolution feedforward network subunit includes a convolution kernel, a normalization layer, a convolution kernel, an activation function layer and a convolution kernel connected in sequence in a residual manner; The lightweight multi-head attention subunit includes three branches, each of which includes a convolution kernel and a linear layer; Among them, the second convolutional layer and the fourth convolutional layer have the same structure as the first convolutional layer, and the fifth convolutional layer has the same structure as the third convolutional layer.

7. The method for training a grain recognition model according to claim 4, characterized in that: The text feature extraction submodule includes multiple conversion encoder units, each of which is connected to a normalization layer, and the conversion encoder unit includes a multi-head attention subunit and a feedforward neural network subunit connected in sequence in a residual manner.

8. The method for training a grain recognition model according to claim 4, characterized in that: The image projection mapping submodule and the text projection mapping submodule include fully connected layers.

9. A method for identifying grains, characterized in that: The method comprises: Get the grain image; The grain image is input into a trained grain recognition model for recognition to obtain a recognition result of the grain image, wherein the recognition result includes the grain type and physical properties, and the trained grain recognition model is trained according to the grain recognition model training method according to any one of claims 1-8.

10. The grain identification method according to claim 9, characterized in that: Before inputting the grain image into the trained grain recognition model for recognition to obtain the recognition result of the grain image, the method further includes: Performing a simplified operation on the trained grain recognition model; The simplification operation includes at least one of the following: removing other branch structures except the first branch in the convolution layer, and removing the normalization layer and the residual structure in the local perception subunit.

11. The grain identification method according to claim 9, characterized in that: The method further comprises: Using the text encoding module in the trained grain recognition model to extract features from multiple grain text description words, multiple text feature vector data are obtained; Among them, multiple text feature vector data that have been feature extracted in advance are stored in a storage unit corresponding to the trained grain recognition model, so as to calculate the similarity score between the image feature vector data and the text feature vector data, and obtain the recognition result of the grain image based on the similarity score.

12. An electronic device, characterized in that: It comprises a memory and a processor, the memory stores a computer program, and is characterized in that when the processor executes the computer program, it implements the steps of the training method of the grain identification model described in any one of claims 1-8, or implements the steps of the grain identification method described in any one of claims 9-11.

13. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the steps of the training method of the grain identification model described in any one of claims 1-8 are implemented, or the steps of the grain identification method described in any one of claims 9-11 are implemented.

Images