Knowledge distillation-based lightweight agricultural multi-modal named entity recognition system and method

Abstract

The invention discloses a lightweight agricultural multi-modal named entity recognition system and method based on knowledge distillation. The system comprises a data acquisition module, a model construction module, a model lightweight module and a recognition module. The data acquisition module is used for acquiring a multi-modal named entity recognition data set in the agricultural field; the model construction module is used for constructing an FCRCL model and constructing an LSTM-FCRCL model based on the FCRCL model; the model lightweight module is used for training an FCRCL model and an LSTM-FCRCL model by adopting knowledge distillation on the basis of a multi-modal named entity recognition data set to obtain a Tiny-FCRCL model; and the identification module performs multi-modal named entity identification based on the Tiny-FCRCL model to obtain an identification result. The performance of the Tiny-FCRCL model is obviously superior to that of a student model and is close to that of a teacher model.

Description

Technical Field

[0001] This invention belongs to the fields of agriculture and knowledge engineering technology, specifically relating to a lightweight agricultural multimodal named entity recognition system and method based on knowledge distillation. Background Technology

[0002] The rapid advancement of internet technology has greatly enriched information resources, while simultaneously driving an explosive growth in data volume. This vast amount of online information contains a wealth of valuable data, which can be extracted using information extraction techniques in Natural Language Processing (NLP). Named Entity Recognition (NER), a fundamental task in information extraction, aims to identify specific types of entities, such as names of people, places, and organizations, from unstructured text. In the agricultural sector, NER technology can be used to extract key information about crops, pests, pesticides, and agricultural techniques, providing strong support for building agricultural knowledge graphs, intelligent question answering, and large-scale agricultural models. As agricultural news articles increasingly incorporate cutting-edge agricultural knowledge, popular science content, and local characteristics, exploring the application of multimodal learning in agricultural named entity recognition not only helps enrich and develop multimodal learning theories and methods but also provides a more reliable foundation for building multimodal agricultural knowledge graphs, intelligent question answering, and information retrieval applications, promoting the deep application of artificial intelligence in agriculture and contributing to the modernization of agriculture.

[0003] The growth in data volume has also led to a continuous enrichment of data formats. The generated data is no longer limited to text; the proportion of images, videos, and audio data has increased significantly. Against this backdrop, multimodal named entity recognition (MNER) has begun to receive widespread attention. This task not only relies on text content but also integrates data from multiple other modalities such as images and audio to achieve semantic complementarity and information interaction between different modalities, thereby enabling more accurate identification of target entities. When textual information is insufficient, image information can effectively assist in identifying ambiguous entity categories.

[0004] While existing models can achieve excellent accuracy, it has been found during training that their large number of parameters and complex network structure result in high GPU resource consumption and long training times, thus placing high demands on the training environment. Therefore, deployment in low-resource environments has become an important research area. Summary of the Invention

[0005] This invention aims to address the shortcomings of existing technologies and provides the following solutions:

[0006] A lightweight agricultural multimodal named entity recognition system based on knowledge distillation includes: a data acquisition module, a model building module, a model lightweighting module, and a recognition module;

[0007] The data acquisition module is used to acquire a multimodal named entity recognition dataset in the agricultural field;

[0008] The model building module is used to build an FCRCL model and, based on the FCRCL model, to build an LSTM-FCRCL model;

[0009] The model lightweighting module trains the FCRCL model and the LSTM-FCRCL model using knowledge distillation based on the multimodal named entity recognition dataset to obtain the Tiny-FCRCL model.

[0010] The recognition module performs multimodal named entity recognition based on the Tiny-FCRCL model to obtain the recognition result.

[0011] Preferably, the workflow of the data acquisition module includes:

[0012] Build a data collection platform to collect image and text data from agricultural photo news as raw multimodal named entity recognition data;

[0013] The original multimodal named entity recognition data is labeled to obtain the labeled dataset;

[0014] The labeled dataset is statistically analyzed and cleaned to obtain the multimodal named entity recognition dataset.

[0015] Preferably, the FCRCL model includes: a feature extraction layer, a cross-modal alignment auxiliary task, a multimodal feature fusion layer, and an entity label decoding layer;

[0016] The feature extraction layer includes a BERT sublayer and a ResNet-152 sublayer. The BERT sublayer is used to extract text features from the agricultural multimodal named entity recognition dataset, and the ResNet-152 sublayer is used to extract image features from the agricultural multimodal named entity recognition dataset.

[0017] The cross-modal alignment assistance task uses contrastive learning to bring the text features and the image features closer together in space;

[0018] The multimodal feature fusion layer is used to fuse the text features and the image features to obtain multimodal fused features;

[0019] The entity label decoding layer is used to decode the labels of the multimodal fusion features and perform label prediction.

[0020] Preferably, in the model building module, the process of building the LSTM-FCRCL model includes: based on the FCRCL model, replacing the BERT sublayer with a Bi-LSTM sublayer to obtain the LSTM-FCRCL model.

[0021] Preferably, the workflow of the model lightweighting module includes:

[0022] The FCRCL model is trained based on the multimodal named entity recognition dataset, and the first soft prediction result is obtained using the trained teacher model. The LSTM-FCRCL model is trained based on the multimodal named entity recognition dataset, and the second soft prediction result is obtained using the trained student model.

[0023] The cross-entropy is calculated on the first soft prediction result and the second soft prediction result to obtain the soft loss of the distillation process;

[0024] Based on the soft loss, the hard prediction result of the trained student model is obtained. The cross-entropy is calculated between the hard label and the hard prediction result to obtain the hard loss of the distillation process.

[0025] The soft loss and the hard loss are weighted and summed to obtain the total loss of the distillation process. The total loss is then backpropagated to the LSTM-FCRCL model to complete the model lightweighting, resulting in the Tiny-FCRCL model.

[0026] This invention also provides a lightweight agricultural multimodal named entity recognition method based on knowledge distillation. The method, applied to the aforementioned system, includes the following steps:

[0027] Obtain a multimodal named entity recognition dataset in the agricultural field;

[0028] Construct a FCRCL model, and then construct an LSTM-FCRCL model based on the FCRCL model;

[0029] Based on the multimodal named entity recognition dataset, knowledge distillation is used to train the FCRCL model and the LSTM-FCRCL model to obtain the Tiny-FCRCL model;

[0030] Multimodal named entity recognition was performed based on the Tiny-FCRCL model, and the recognition results were obtained.

[0031] Preferably, the method for obtaining the multimodal named entity recognition dataset includes:

[0032] Build a data collection platform to collect image and text data from agricultural photo news as raw multimodal named entity recognition data;

[0033] The original multimodal named entity recognition data is labeled to obtain the labeled dataset;

[0034] The labeled dataset is statistically analyzed and cleaned to obtain the multimodal named entity recognition dataset.

[0035] Preferably, the FCRCL model includes: a feature extraction layer, a cross-modal alignment auxiliary task, a multimodal feature fusion layer, and an entity label decoding layer;

[0036] The feature extraction layer includes a BERT sublayer and a ResNet-152 sublayer. The BERT sublayer is used to extract text features from the agricultural multimodal named entity recognition dataset, and the ResNet-152 sublayer is used to extract image features from the agricultural multimodal named entity recognition dataset.

[0037] The cross-modal alignment assistance task uses contrastive learning to bring the text features and the image features closer together in space;

[0038] The multimodal feature fusion layer is used to fuse the text features and the image features to obtain multimodal fused features;

[0039] The entity label decoding layer is used to decode the labels of the multimodal fusion features and perform label prediction.

[0040] Preferably, the method for constructing the LSTM-FCRCL model includes: based on the FCRCL model, replacing the BERT sublayer with a Bi-LSTM sublayer to obtain the LSTM-FCRCL model.

[0041] Preferably, the method for obtaining the Tiny-FCRCL model includes:

[0042] The FCRCL model is trained based on the multimodal named entity recognition dataset, and the first soft prediction result is obtained using the trained teacher model. The LSTM-FCRCL model is trained based on the multimodal named entity recognition dataset, and the second soft prediction result is obtained using the trained student model.

[0043] The cross-entropy is calculated on the first soft prediction result and the second soft prediction result to obtain the soft loss of the distillation process;

[0044] Based on the soft loss, the hard prediction result of the trained student model is obtained. The cross-entropy is calculated between the hard label and the hard prediction result to obtain the hard loss of the distillation process.

[0045] The soft loss and the hard loss are weighted and summed to obtain the total loss of the distillation process. The total loss is then backpropagated to the LSTM-FCRCL model to complete the model lightweighting, resulting in the Tiny-FCRCL model.

[0046] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0047] This invention addresses the issues of high training resource requirements and long training time associated with the FCRCL model by introducing a knowledge distillation method to lightweight the FCRCL model, resulting in the Tiny-FCRCL model. In the knowledge distillation process, the FCRCL model is used as the teacher model, and the LSTM-FCRCL model as the student model. The teacher model guides the student model during training, improving the student model's performance and generalization ability while maintaining its lightweight characteristics. Analysis of experimental results on the FD-CH dataset shows that Tiny-FCRCL reduces the number of parameters by 32% compared to the FCRCL model, reduces average training time by 30%, and reduces average GPU memory usage by 33%. Results from precision, recall, and F1 score demonstrate that the Tiny-FCRCL model significantly outperforms the student model (LSTM-FCRCL) and approaches the performance of the teacher model (FCRCL). This approach ensures that the model maintains both lightweight advantages and relatively excellent multimodal named entity recognition performance. Attached Figure Description

[0048] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This is a schematic diagram of the system structure according to an embodiment of the present invention;

[0050] Figure 2 This is a schematic diagram of the FCRCL model structure according to an embodiment of the present invention;

[0051] Figure 3 This is a schematic diagram of the LSTM-FCRCL model structure according to an embodiment of the present invention;

[0052] Figure 4 This is a schematic diagram of the knowledge distillation process according to an embodiment of the present invention. Detailed Implementation

[0053] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0055] Example 1

[0056] In this embodiment, as Figure 1 As shown, a lightweight agricultural multimodal named entity recognition system based on knowledge distillation includes: a data acquisition module, a model building module, a model lightweighting module, and a recognition module.

[0057] The data acquisition module is used to acquire multimodal named entity recognition datasets in the agricultural field.

[0058] The workflow of the data acquisition module includes: building a data collection platform to collect text and image data from agricultural photo news as raw multimodal named entity recognition data; labeling the raw multimodal named entity recognition data to obtain a labeled dataset; performing statistical analysis and data cleaning on the labeled dataset to obtain a multimodal named entity recognition dataset.

[0059] In this embodiment, firstly, a Python web crawler script is written to crawl the URLs of the "Photo News" section of the *Farmers' Daily* from 2021 to 2023 and save them to a CSV file. Then, because the web crawler cannot parse the specific text content of the "Photo News" section and contains some unwanted dirty data text (such as "Photographer: Li xx", "Reporter: Qi xx", and some line breaks and spaces), this embodiment uses PyQT to build a data collection platform to facilitate the acquisition and processing of raw data. The data collection platform reads all URLs from the CSV file and writes the images in the form of "IMGID:Image ID" (e.g., IMGID:0001) into a line in the raw data file data.txt. All the obtained text content is manually edited on the platform, deleting unwanted dirty data before saving. The text content will be written to the corresponding "IMGID:Image ID" line, separated by spaces. Finally, such a line of data constitutes a complete piece of raw multimodal named entity recognition data.

[0060] The original multimodal named entity recognition data was then labeled using the BIO format to obtain the labeled dataset, with the image ID as the first row of the BIO format data. In the original multimodal named entity recognition data, except for a few data points, the data format generally conforms to "xxx location / xxx person from xxx organization performing a certain action on xxx crop / cattle, sheep, fish, shrimp, and other agricultural products". Therefore, five types of entities are defined: Person (PER), Location (LOC), Organization (ORG), Production (PROD), and Miscellaneous (MISC). Each entity type and its corresponding label are listed, and data examples for each type are provided.

[0061] After statistical analysis of the labeled dataset and final data cleaning, a multimodal named entity recognition dataset was obtained. The constructed dataset consists of 4,000 image-text pairs, which are divided into training, validation, and test sets in an 8:1:1 ratio. Each file contains 3,200, 400, and 400 data entries, respectively. Further entity counts are performed on each data file.

[0062] The model building module is used to build the FCRCL model and then build the LSTM-FCRCL model based on the FCRCL model.

[0063] In this embodiment, the overall structure of the proposed Multimodal Named Entity Recognition model Based on Feature Cross Replacement and Contrast Learning (FCRCL) is as follows: Figure 2 As shown.

[0064] The FCRCL model includes: a feature extraction layer, a cross-modal alignment auxiliary task, a multimodal feature fusion layer, and an entity label decoding layer.

[0065] The feature extraction layer includes a BERT sublayer and a ResNet-152 sublayer. The BERT sublayer is used to extract text features from the multimodal named entity recognition dataset in the agricultural field, and the ResNet-152 sublayer is used to extract image features from the multimodal named entity recognition dataset in the agricultural field.

[0066] In this embodiment, BERT sublayers are used to extract text features, fully capturing the semantic information of the text. Image features are extracted using ResNet-152; its multi-layered network structure enables it to acquire richer and more detailed features. The output of the last hidden layer of the BERT sublayer is used to represent text features; the dimension of the hidden layer is typically set to 768 dimensions (which can be adjusted according to experimental needs). Specifically, for a text S = {s0, s1, ..., s...}, ... n+1}, S represents a word, s0 corresponds to [CLS], s n+1 Corresponding to [SEP], n represents the length of the text sequence, and the maximum length of the text sequence is set to 256. The final output text feature representation is T = {t0, t1, ..., t2}. n+1 The ResNet-152 sublayer first takes an image from a text-image pair in the dataset as input, and performs a center crop to resize the image, resulting in a 3×224×224 pixel input (3 channels). Then, the ResNet-152 sublayer, with the fully connected and pooling layers removed, calculates the feature representation of the image. Since the maximum length of the text sequence is set to 256, adaptive pooling is used to adjust the feature map to 16×16, resulting in 256 feature vectors, represented as R = (r1, r2, ..., r...). 256 Finally, a fully connected layer is used to map the 2048-dimensional image vector to a 768-dimensional vector.

[0067] Cross-modal alignment assistance tasks bring text features and image features closer together in space through contrastive learning.

[0068] The goal of the cross-modal alignment auxiliary task is to narrow the gap between text and image representations in the representation space through contrastive learning, making the text and image representations in the FCRCL model more similar. This process brings semantically similar image and text representations closer together, while semantically dissimilar image and text representations become more distant, effectively reducing the semantic differences between images and text and improving cross-modal alignment performance. The entire auxiliary task can be broadly divided into two parts: constructing positive and negative samples for contrastive learning, and calculating the similarity between images and text.

[0069] The multimodal feature fusion layer is used to fuse text features and image features to obtain multimodal fused features.

[0070] This embodiment proposes a multimodal feature fusion layer based on feature cross-substitution, using two Transformer encoders with shared parameters as the backbone model. [CLS] (a special marker in Transformer used to derive sentence-level embeddings) is inserted at the beginning of the generated text and image embedding vectors. Then, cross-substitution is performed before the feedforward network (FNN) layer in the Transformer encoder to ensure better gradient propagation, reduce the risk of gradient explosion or vanishing, make deep models easier to train and converge, and ensure a more suitable feature distribution when combined with residual connections, thus better utilizing the "identity mapping" advantage and ensuring that information can be directly transmitted without loss. To learn the global representation of the input vector, this module does not perform cross-substitution operations in shallow layers, but only performs cross-substitution operations between the two modalities within the cross-substitution interval for deep feature fusion. Specifically, the module introduces two hyperparameter replacement start layers (RSL) and replacement end layers (REL) to control the start and end layers of information exchange, respectively. In each exchange layer, the feature with the minimum attention score compared to the global feature representation [CLS] in one modality is selected for labeling, and the feature to be replaced is replaced with the average embedding of all labels in the other modality.

[0071] The entity label decoding layer is used to decode labels from multimodal fusion features and perform label prediction.

[0072] In this embodiment, after multimodal feature fusion, the fused features are fed into a Conditional Random Field (CRF). This layer uses the CRF algorithm to find the optimal label sequence by considering the dependencies between labels. By analyzing the interaction between multimodal data and labels, the teacher FCRCL model can accurately predict labels.

[0073] The process of building the LSTM-FCRCL model in the model building module includes: replacing the BERT sublayer with a Bi-LSTM sublayer based on the FCRCL model to obtain the LSTM-FCRCL model.

[0074] In this embodiment, the LSTM-FCRCL model is a lightweight model based on the teacher FCRCL model, using Bi-LSTM sublayers to replace BERT sublayers as the text feature extraction network. The student LSTM-FCRCL model employs a lightweight Bi-LSTM network, exhibiting low computational overhead during training, including time, memory usage, and GPU resource consumption, thus demonstrating good feasibility in practical deployment. However, at the same time, due to the simple nature of its model structure, it has certain limitations in learning complex features and mining deep semantic information, and its generalization ability is relatively weak.

[0075] To address the aforementioned issues, this invention employs LSTM-FCRCL as the student model and FCRCL as the teacher model in its knowledge distillation strategy. This choice helps transfer deep semantic knowledge from the teacher model to the student model, thereby improving its learning performance to some extent and overcoming the limitations caused by model simplification. Overall, the LSTM-FCRCL model is not only structurally simple but also capable of extracting a large amount of valuable knowledge from FCRCL, giving it a significant advantage in lightweight scenarios. The specific structure of the LSTM-FCRCL model is as follows: Figure 3 As shown.

[0076] For a piece of image and text data, the text S = {s1, s2, ..., s...} i}, where s i This represents the vector representation of the i-th word in the text, and sets the maximum length of the text sequence to 256. Any length exceeding this will be truncated; otherwise, the length of the text sequence will be adjusted. <pad>filling, <pad>It doesn't participate in the computation; it's only used for alignment with the feature dimensions of the image modality. After S is input into the Bi-LSTM, it computes LSTMs in both the left and right directions, with the left-side LSTM being the only one involved. Calculation method, to the right The calculation method is as follows:

[0077]

[0078] in, and This represents the text encoding obtained by LSTM computation in both the left and right directions. Indicates the output of the next leftward state. This indicates the output of the previous rightward state. (Through both directions) and The calculated encoded representation and It captures long-distance correlation information more effectively and possesses good context awareness. Finally, the encoded representations from the two directions are concatenated to obtain the final output of the Bi-LSTM, calculated as follows:

[0079]

[0080] Where [;] denotes the vector concatenation operation, T represents the final encoded representation, and T∈R n×d It is in the same dimension as image features.

[0081] The workflow of the multimodal feature fusion layer in the student LSTM-FCRCL model includes: First, since the input text features T and image features I are not in the same dimensional space, the image representation obtained by the ResNet-152 sublayer is [batch_size, 16, 16, 768], and the text representation obtained by the BERT sublayer is [batch_size, 256, 768]. Therefore, a flattening operation is needed to bring the two modalities into the same dimensional space. The calculation method is as follows:

[0082]

[0083] in, This represents the text features in the same dimensional space. Then, a global representation T is added at the beginning of both. cls and I cls T cls and I cls These are tensor elements constructed based on the dimensions of text features T and image features I. They can expand a vector to the same batch size as the input tensor to allow for element-wise operations in subsequent computations. The specific calculation method is as follows:

[0084]

[0085] in, and This represents the text and image feature representations obtained after concatenation. and Next, the multi-head attention of both needs to be calculated, as shown below:

[0086]

[0087] in, The scaling factor is used. After Softmax calculation, an attention map matrix is ​​generated, which represents the attention scores between each label. The first row of the attention map represents the attention scores of [CLS] for all labels in the input. This embodiment only uses a Transformer encoder, where each layer first calculates multi-head self-attention, followed by an addition and normalization layer, a feedforward network layer, and a final addition and normalization layer, ultimately obtaining... and Where i∈[0,5] represents the current layer number in the multimodal feature fusion layer based on feature cross-substitution. Then, during the cross-substitution process, the average value of the image and text modal features needs to be calculated, as shown below:

[0088]

[0089] Where d represents the dimension of the feature vector, and n represents the length of the text sequence. After obtaining the average feature values ​​of the text and the image, the text feature replacement process iterates through the image feature attention scores. For values ​​smaller than the global feature attention score of the text, the average feature value of the current layer image is used. Similarly, the same replacement process was applied to image features. To minimize the loss of feature information caused by the replacement, this invention employs residual connections during the replacement process, as shown below:

[0090]

[0091] Where k represents the index of the selected attention score value that is less than the global feature value; [k,:] represents the index of all values ​​less than the global feature value; and i represents the cross-replacement layer number. Finally, after all replacement and non-replacement layers, the feature representation of the swapped text is obtained. and image feature representation The final fusion is achieved through a gating mechanism, and the calculation method is as follows:

[0092]

[0093] Where M represents the final fused multimodal feature representation, Gate represents the gating mechanism, G represents the result calculated by the gating mechanism, and [:,1,:] represents removing the [CLS] concatenated at the beginning.

[0094] The workflow of the contrastive learning-based cross-modal alignment auxiliary task in the student LSTM-FCRCL model includes: after feature extraction, it is necessary to compute the global representation T of the text. g and the global representation of the image I g Positive and negative samples are constructed from a batch of data. Positive samples consist of corresponding images and text pairs within the batch, with their representations as close as possible. Negative samples enumerate non-corresponding image and text pairs within the batch, with their representations as far apart as possible. The teacher's FCRCL model is then optimized using a contrastive loss function, resulting in smaller feature representation distances for positive samples and larger feature representation distances for negative samples. Taking a batch containing four data points as an example, the positive and negative samples within the batch can be constructed using the above method, as shown in Table 1.

[0095] Table 1

[0096]

[0097] In the table above, for an input with a batch size of N, This represents a corresponding image-text pair in the batch, and is also a representation of a positive sample. And j≠i, indicating the composition of negative samples. For a batch of input, the number of positive samples is N, and the negative samples are all combinations of non-corresponding image-text pairs, containing a total of (N×(N-1)) negative samples. Even though there may be positive samples with not closely related representations in all image-text pairs, such a small number of poorly matched positive samples will not have a decisive impact on the final result under large-scale data. The number of negative samples has a positive correlation with the overall performance of contrastive learning. After constructing the positive and negative samples, it is necessary to calculate the similarity between the image representation and the text representation. Similarity aims to measure the correlation between the two modalities. This embodiment uses cosine similarity to calculate the similarity between the two modalities. The calculation methods for the similarity of positive samples and the similarity of negative samples are as follows:

[0098]

[0099] To improve the similarity between images and text, this embodiment calculates the contrast loss in two directions using both image-to-text and text-to-image similarity methods, thus optimizing the final loss. The specific calculation methods for the two loss functions are as follows:

[0100]

[0101]

[0102] Where N represents the number of data points in a batch, and τ represents the temperature parameter, which is a hyperparameter. Finally, the final loss L for the cross-modal alignment auxiliary task... AUX The contrast loss from both the image-to-text and text-to-image directions is weighted and summed, as shown below:

[0103]

[0104] Here, λ represents a hyperparameter, λ∈[0,1].

[0105] The workflow of the entity label decoding layer in the student LSTM-FCRCL model includes: inputting the final concatenated multimodal feature sequence M into the CRF layer, and using the original text T and the corresponding image I to predict the probability of the final label result sequence y. The calculation method is as follows:

[0106]

[0107] Where E(m,y) is the score value composed of the launch score and the transfer score, and y * ∈Y * Let L represent all possible combinations of label sequences. Then, the loss L for the final multimodal named entity recognition main task is obtained by calculating the minimization of the negative log-like natural function. MNER The specific calculations are as follows:

[0108]

[0109] Finally, the overall loss function for model training consists of the cross-modal alignment-assisted loss L. AUX And the loss function of the main task L MNER The weighted average is obtained, and the calculation method is as follows:

[0110] L=αL AUX +(1-α)L MNER

[0111] Here, α represents a hyperparameter used to adjust the weights between the two losses.

[0112] The lightweight model module is based on a multimodal named entity recognition dataset. It uses knowledge distillation to train the FCRCL model and the LSTM-FCRCL model to obtain the Tiny-FCRCL model.

[0113] The workflow of the model lightweighting module includes: training the FCRCL model based on the multimodal named entity recognition dataset, and obtaining the first soft prediction result using the trained teacher model; training the LSTM-FCRCL model based on the multimodal named entity recognition dataset, and obtaining the second soft prediction result using the trained student model; calculating the cross-entropy between the first and second soft prediction results to obtain the soft loss of the distillation process; obtaining the hard prediction result of the trained student model based on the soft loss, calculating the cross-entropy between the hard label and the hard prediction result to obtain the hard loss of the distillation process; weighted summing of the soft loss and hard loss to obtain the total loss of the distillation process, and backpropagating the total loss to the LSTM-FCRCL model to complete the model lightweighting and obtain the Tiny-FCRCL model.

[0114] In this embodiment, as Figure 4 As shown, in the knowledge distillation process, the teacher FCRCL model is first trained using a multimodal named entity recognition dataset to obtain a high-performance teacher FCRCL model. Then, the final network output e of the entity decoding layer of the trained teacher FCRCL model is selected. i The model is then warmed using Softmax. After this warming process, the output entity class probabilities are smoother, allowing the soft predictions to incorporate more semantic information. The soft prediction result e of the trained teacher FCRCL model is shown below. i The calculation method for _soft_prediction is as follows:

[0115]

[0116] After obtaining the soft predictions from the teacher model, the student LSTM-FCRCL model is trained. First, a warming process is applied using Softmax (T = t) to obtain the soft predictions from the trained student LSTM-FCRCL model. Then, the cross-entropy is calculated between the soft predictions from the teacher model and the student LSTM-FCRCL model to obtain the soft loss L of the distillation process. soft The calculation method is as follows:

[0117] L soft =CE(T) soft S soft )

[0118] Where CE represents the cross-entropy loss function, and T soft S represents the soft prediction result of the FCRCL teacher model. soft This represents the soft prediction results of the student LSTM-FCRCL model. The soft loss L during the distillation process is obtained. soft Then, the hard prediction results of the student LSTM-FCRCL model are obtained using the Softmax (T=1) operation. The hard loss L in the distillation process is obtained by calculating the cross-entropy loss from the hard labels and hard prediction results. hard The calculation method is as follows:

[0119] L hard =CE(S) hard Target hard )

[0120] Among them, S hard This represents the hard prediction result of the student LSTM-FCRCL model, Target hard This represents a hard label. Finally, regarding the soft loss L... soft and hard loss L hard The total loss L of the distillation process is obtained by weighted summation. total The calculation method is as follows:

[0121] L total =λL soft +(1-λ)L hard

[0122] Here, λ represents a hyperparameter, λ∈[0,1]. The total loss L... total The knowledge distillation process is then completed by backpropagating to the student model, resulting in the Tiny-FCRCL model.

[0123] The recognition module performs multimodal named entity recognition based on the Tiny-FCRCL model and obtains the recognition results.

[0124] Example 2

[0125] In this embodiment, a lightweight agricultural multimodal named entity recognition method based on knowledge distillation includes the following steps:

[0126] S1. Obtain a multimodal named entity recognition dataset in the agricultural field.

[0127] The method for obtaining a multimodal named entity recognition dataset includes: building a data collection platform to collect image and text data from agricultural photo news as raw multimodal named entity recognition data; labeling the raw multimodal named entity recognition data to obtain a labeled dataset; performing statistical analysis and data cleaning on the labeled dataset to obtain a multimodal named entity recognition dataset.

[0128] S2. Construct the FCRCL model and then construct the LSTM-FCRCL model based on the FCRCL model.

[0129] The FCRCL model comprises: a feature extraction layer, a cross-modal alignment auxiliary task, a multimodal feature fusion layer, and an entity label decoding layer. The feature extraction layer includes BERT sub-layers and ResNet-152 sub-layers. The BERT sub-layer is used to extract text features from the agricultural multimodal named entity recognition dataset, and the ResNet-152 sub-layer is used to extract image features from the agricultural multimodal named entity recognition dataset. The cross-modal alignment auxiliary task uses contrastive learning to bring text features and image features closer together in space. The multimodal feature fusion layer is used to fuse text features and image features to obtain multimodal fused features. The entity label decoding layer is used to decode the labels of the multimodal fused features and perform label prediction.

[0130] The method for constructing an LSTM-FCRCL model includes: based on the FCRCL model, replacing the BERT sublayer with a Bi-LSTM sublayer to obtain the LSTM-FCRCL model.

[0131] S3. Based on the multimodal named entity recognition dataset, knowledge distillation is used to train the FCRCL model and the LSTM-FCRCL model to obtain the Tiny-FCRCL model.

[0132] The method for obtaining the Tiny-FCRCL model includes: training the FCRCL model on a multimodal named entity recognition dataset and obtaining the first soft prediction result using the trained teacher model; training the LSTM-FCRCL model on the multimodal named entity recognition dataset and obtaining the second soft prediction result using the trained student model; calculating the cross-entropy between the first and second soft prediction results to obtain the soft loss of the distillation process; obtaining the hard prediction result of the trained student model based on the soft loss, calculating the cross-entropy between the hard label and the hard prediction result to obtain the hard loss of the distillation process; weighted summing of the soft loss and hard loss to obtain the total loss of the distillation process, and backpropagating the total loss to the LSTM-FCRCL model to complete model lightweighting and obtain the Tiny-FCRCL model.

[0133] S4. Multimodal named entity recognition is performed based on the Tiny-FCRCL model to obtain the recognition results.

[0134] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.< / pad> < / pad>

Claims

1. A lightweight agricultural multi-modal named entity recognition system based on knowledge distillation, characterized in that, The application relates to a method for multi-modal named entity recognition in the agricultural field. The method comprises the following steps: a data acquisition module, a model construction module, a model lightweight module and an identification module; the data acquisition module is used for acquiring a multi-modal named entity recognition data set in the agricultural field; the model construction module is used for constructing an FCRCL model and constructing an LSTM-FCRCL model based on the FCRCL model; the model lightweight module is used for training the FCRCL model and the LSTM-FCRCL model by adopting knowledge distillation based on the multi-modal named entity recognition data set, so as to obtain a Tiny-FCRCL model; 2. The knowledge distillation based lightweight agricultural multi-modal named entity recognition system according to claim 1, wherein, the identification module is used for performing multi-modal named entity recognition based on the Tiny-FCRCL model, so as to obtain an identification result. The working process of the data acquisition module comprises the following steps: a data collection platform is constructed, and picture-text data in agricultural picture news is collected as original multi-modal named entity recognition data; the original multi-modal named entity recognition data is labeled to obtain a labeled data set; 3. The knowledge distillation based lightweight agricultural multi-modal named entity recognition system according to claim 1, wherein, statistical analysis is performed on the labeled data set, and data cleaning is performed, so as to obtain the multi-modal named entity recognition data set. The FCRCL model comprises a feature extraction layer, a cross-modal alignment auxiliary task, a multi-modal feature fusion layer and an entity label decoding layer; the feature extraction layer comprises a BERT sublayer and a ResNet-152 sublayer, the BERT sublayer is used for extracting text features in the multi-modal named entity recognition data set in the agricultural field, and the ResNet-152 sublayer is used for extracting image features in the multi-modal named entity recognition data set in the agricultural field; the cross-modal alignment auxiliary task is used for narrowing the text features and the image features in space through contrast learning; the multi-modal feature fusion layer is used for fusing the text features and the image features, so as to obtain multi-modal fusion features; 4. The knowledge distillation based lightweight agricultural multi-modal named entity recognition system according to claim 3, characterized in that, the entity label decoding layer is used for label decoding of the multi-modal fusion features and label prediction.

5. The knowledge distillation based lightweight agricultural multi-modal named entity recognition system according to claim 1, wherein, In the model construction module, the process of constructing the LSTM-FCRCL model comprises the following steps: the BERT sublayer is replaced by a Bi-LSTM sublayer based on the FCRCL model, so as to obtain the LSTM-FCRCL model. The working process of the model lightweight module comprises the following steps: the FCRCL model is trained based on the multi-modal named entity recognition data set, and a first soft prediction result is obtained by using a trained teacher model; the LSTM-FCRCL model is trained based on the multi-modal named entity recognition data set, and a second soft prediction result is obtained by using a trained student model; cross entropy is calculated for the first soft prediction result and the second soft prediction result, so as to obtain a soft loss in a distillation process; a hard prediction result of the trained student model is obtained based on the soft loss, cross entropy is calculated for a hard label and the hard prediction result, so as to obtain a hard loss in the distillation process; The soft loss and the hard loss are weighted and summed to obtain a total loss of a distillation process, and the total loss is back-propagated to the LSTM-FCRCL model to complete model lightening to obtain the Tiny-FCRCL model.

6. A method for lightweight agricultural multi-modal named entity recognition based on knowledge distillation, applied to the system of any one of claims 1-5, characterized in that, The method comprises the following steps: An agricultural multi-modal named entity recognition dataset is obtained; An FCRCL model is constructed, and an LSTM-FCRCL model is constructed based on the FCRCL model; The FCRCL model and the LSTM-FCRCL model are trained based on the multi-modal named entity recognition dataset by using knowledge distillation to obtain a Tiny-FCRCL model; Multi-modal named entity recognition is performed based on the Tiny-FCRCL model to obtain a recognition result.

7. The method of claim 6, wherein the method is based on knowledge distillation. The method for obtaining the multi-modal named entity recognition dataset comprises: A data collection platform is constructed to collect image-text data in agricultural picture news as original multi-modal named entity recognition data; The original multi-modal named entity recognition data is labeled to obtain a labeled dataset; The labeled dataset is statistically analyzed and cleaned to obtain the multi-modal named entity recognition dataset.

8. The method of claim 6, wherein the method is based on knowledge distillation. The FCRCL model comprises a feature extraction layer, a cross-modal alignment auxiliary task, a multi-modal feature fusion layer and an entity label decoding layer; The feature extraction layer comprises a BERT sublayer and a ResNet-152 sublayer, the BERT sublayer is used to extract text features in the agricultural multi-modal named entity recognition dataset, and the ResNet-152 sublayer is used to extract image features in the agricultural multi-modal named entity recognition dataset; The cross-modal alignment auxiliary task narrows the text features and the image features in space through contrast learning; The multi-modal feature fusion layer is used to fuse the text features and the image features to obtain multi-modal fusion features; The entity label decoding layer is used to decode the multi-modal fusion features to perform label prediction.

9. The method of claim 8, wherein the method is based on knowledge distillation. The method for constructing the LSTM-FCRCL model comprises: using a Bi-LSTM sublayer to replace the BERT sublayer based on the FCRCL model to obtain the LSTM-FCRCL model.

10. The method of claim 6, wherein the method is based on knowledge distillation for lightweight agricultural multi-modal named entity recognition. The method for obtaining the Tiny-FCRCL model comprises: The FCRCL model is trained based on the multi-modal named entity recognition dataset, and a first soft prediction result is obtained by using a trained teacher model, the LSTM-FCRCL model is trained based on the multi-modal named entity recognition dataset, and a second soft prediction result is obtained by using a trained student model; Cross entropy is calculated for the first soft prediction result and the second soft prediction result to obtain a soft loss of a distillation process; A hard prediction result of the trained student model is obtained based on the soft loss, and cross entropy is calculated for a hard label and the hard prediction result to obtain a hard loss of the distillation process; The soft loss and the hard loss are weighted and summed to obtain a total loss of a distillation process, and the total loss is back-propagated to the LSTM-FCRCL model to complete model lightening to obtain the Tiny-FCRCL model.

Images