Method and apparatus for generating agricultural video tags using multi-resolution analysis

Abstract

This invention discloses a method and apparatus for generating agricultural video tags using multi-resolution analysis. The method includes: obtaining a frame vector carrying location encoding from an agricultural video to be detected; inputting the frame vector carrying location encoding into a multi-layer encoder; inputting a multi-granularity agricultural video encoding vector into a multi-layer decoder; using the decoding output of each decoder layer and the encoding output of each encoder layer in the multi-layer encoder as input to the next decoder layer; finally obtaining a final decoded feature vector after multiple decodings; obtaining multiple pending agricultural video summaries and a confidence level corresponding to each pending agricultural video summary based on the final decoded feature vector; determining a multi-granularity video summary of the agricultural video to be detected based on the confidence level; and obtaining a storage tag for the agricultural video to be detected based on the multi-granularity video summary. This invention maintains high accuracy and generalization ability even when dealing with complex agricultural videos.

Description

Technical Field

[0001] This invention relates to the field of video processing technology, and in particular to a method and apparatus for generating agricultural video tags using multi-resolution analysis. Background Technology

[0002] In the development of data-driven agriculture, video has become an important carrier of agricultural information dissemination, and tags are a key tool for obtaining accurate information and locating data in multiple fields, playing an increasingly important role in crop growth monitoring, pest and disease control, and assessing key behaviors in agricultural production. However, manually analyzing this video data is time-consuming and labor-intensive, and struggles to cope with the ever-increasing volume of data. Therefore, developing an efficient and automated video tag generation technology is of great significance for improving the utilization efficiency and management level of agricultural video data.

[0003] Video tagging is a tool that accurately describes video content and plays a crucial role in the organization and retrieval of agricultural videos. By analyzing multi-dimensional elements such as images, text, voice-over, and behavior in agricultural videos, video tags provide users with a comprehensive description of the video content. They are commonly used in scenarios such as agricultural knowledge graph construction, production process optimization, and crop growth monitoring.

[0004] However, current agricultural video tagging technologies have certain limitations. Specifically, while traditional single-resolution video analysis methods can generate accurate tags in some cases, their accuracy and generalization ability are limited when dealing with complex agricultural videos. Furthermore, existing technologies also have shortcomings in terms of resource consumption, processing speed, and user experience. Summary of the Invention

[0005] To address the aforementioned problems in the existing technology, this invention provides a method and apparatus for generating agricultural video tags using multi-resolution analysis.

[0006] This invention discloses a method for generating agricultural video tags using multi-resolution analysis, comprising:

[0007] Step 1: Obtain the frame vector carrying the location code from the agricultural video to be detected;

[0008] Step 2: Input the frame vector carrying the position encoding into the multi-layer encoder. The object features of different granularities extracted from the agricultural video to be detected and the video summary of the agricultural video to be detected constitute a multi-granularity agricultural video encoding vector. The multi-granularity agricultural video encoding vector is then input into the multi-layer decoder. The decoding output of each layer of the multi-layer decoder and the encoding output of each layer of the multi-layer encoder are used together as the input of the next layer of the decoder. Finally, the final decoded feature vector after multiple decodings is obtained.

[0009] Step 3: Based on the final decoded feature vector, obtain multiple pending agricultural video summaries and the confidence level corresponding to each pending agricultural video summary. Determine the multi-granularity video summaries of the agricultural videos to be detected based on the confidence levels. Based on the multi-granularity video summaries of the agricultural videos to be detected, obtain the storage tags of the agricultural videos to be detected.

[0010] Further, step 1 includes:

[0011] Step 11: Perform frame segmentation on the agricultural video to be detected, extract object features of different granularities from the frame segments, where each frame segment may contain multiple different object features; perform event summary detection on the frame segments of the agricultural video to be detected, and at the same time, use the tag information carried by the object features to complete the detected summary information, and finally obtain the video summary of the corresponding frame segment.

[0012] Step 12: Extract frame vectors containing multi-scale information frame by frame from the agricultural video to be detected, and initialize the frame vectors with position encoding to obtain frame vectors carrying position encoding.

[0013] Further, step 11 includes:

[0014] Step 11: Extract crop features, agricultural meteorological features, agricultural activity features, and agricultural pest features from the agricultural video clips to be detected;

[0015] Step 12: Use the pre-trained PDVC model to perform event summary detection on the agricultural video to be detected frame by frame, and select the summary with the highest confidence as the initial video summary for each frame segment;

[0016] Step 13: Complete the initial video summary with the tag information carried by the object features to finally obtain the video summary of the corresponding frame segment.

[0017] Further, step 11 includes:

[0018] Step 111: Use the vit model pre-trained on the Agriculture crop images dataset to extract crop features for each agricultural video frame segment to be detected. If crop features are extracted, all extracted crop features constitute the crop feature vector to be encoded for each agricultural video frame segment to be detected. Then, use the pre-built crop feature recognition library to label the crop feature vector to be encoded for each agricultural video frame segment to be detected, thereby obtaining the crop feature vector to be encoded for each agricultural video frame segment to be detected carrying crop labels.

[0019] Step 112: Use the ResNet50 model pre-trained on the Multi-class weather dataset to extract agricultural meteorological features for each agricultural video frame segment to be detected. If agricultural meteorological features are extracted, all extracted agricultural meteorological features constitute the agricultural meteorological feature vector to be encoded for each agricultural video frame segment to be detected. Then, use the pre-built agricultural meteorological feature recognition library to label the agricultural meteorological feature vector to be encoded for each agricultural video frame segment to be detected, thereby obtaining the agricultural meteorological feature vector to be encoded for each agricultural video frame segment to be detected carrying agricultural meteorological labels.

[0020] Step 113: Use the vit model pre-trained on the Agricultural-Vision dataset to extract agricultural activity features for each agricultural video frame segment to be detected. If agricultural activity features are extracted, all extracted agricultural activity features constitute the agricultural activity feature vector to be encoded for each agricultural video frame segment to be detected. Then, use the pre-built agricultural activity feature recognition library to label the agricultural activity feature vector to be encoded for each agricultural video frame segment to be detected, thereby obtaining the agricultural activity feature vector to be encoded for each agricultural video frame segment to be detected carrying agricultural activity labels.

[0021] Step 114: Use the YOLO V8 model pre-trained on the IP102 dataset to extract agricultural pest features for each agricultural video frame segment to be detected. If agricultural pest features are extracted, all extracted agricultural pest features constitute the agricultural pest feature vector to be encoded for each agricultural video frame segment to be detected. Then, use the pre-built agricultural pest feature recognition library to label the agricultural pest feature vector to be encoded for each agricultural video frame segment to be detected, thereby obtaining the agricultural pest feature vector to be encoded for each agricultural video frame segment to be detected carrying agricultural pest labels.

[0022] Further, step 2 includes:

[0023] Step 21: Convert the frame vector Z' carrying the position code obtained in Step 1 into a frame vector Z'. mu The lti input is fed into the multi-layer Transformer encoder;

[0024] Step 22: Encode the possible crop feature vectors, agricultural meteorological feature vectors, agricultural activity feature vectors, and agricultural pest feature vectors that may exist in each agricultural video frame segment to be detected into word vectors, and encode the initial video summary of each agricultural video frame segment to be detected into sentence vectors with the same length as the word vectors through sentence encoding;

[0025] Step 23: Combine the sentence vector and word vector obtained in Step 22 to form an encoding vector;

[0026] Step 24: Input the encoded vector obtained in step 23 into the multi-layer Transformer decoder, and at the same time, input the encoded output of the Transformer encoder obtained in step 21 as the supervision target into the current layer Transformer decoder to obtain the decoding output of the current layer Transformer decoder.

[0027] Step 25: Input the output of the current layer's Transformer decoder into the next layer's Transformer decoder, and simultaneously input the encoding output of the next layer's Transformer encoder as the supervision target into the current layer's Transformer decoder. Finally, obtain the output of the current layer's Transformer decoder. After N iterations, obtain the final decoding output; N is the number of encoder-decoder layers.

[0028] Further, step 3 includes:

[0029] Step 31: Based on the final decoded feature vector, obtain multiple pending agricultural video summaries and the confidence level of each pending agricultural video summary. Finally, determine the pending agricultural video summary with the highest confidence level as the multi-granularity video summary of the agricultural video to be detected.

[0030] Step 32: Extract keywords from the multi-granularity video summary of the agricultural video to be detected, and use the extracted keywords as storage tags for the agricultural video to be detected.

[0031] Further, step 31 includes:

[0032] The final decoded feature vectors are input into the confidence feedforward model and the LSTM model with attention mechanism, respectively, to obtain multiple undetermined agricultural video summaries output by the LSTM model and the confidence level of each undetermined agricultural video summary output by the confidence feedforward model. Finally, the undetermined video summary with the highest confidence level is selected as the multi-granularity video summary of the agricultural video to be detected.

[0033] Further, step 32 includes:

[0034] Multi-granular video summary keywords were extracted from agricultural videos to be detected using a RoBERTa-based word vector and global semantic vector extraction model, and the extracted keywords were used as storage tags for the agricultural videos to be detected.

[0035] Furthermore, step 32 specifically includes:

[0036] Step 321: Perform text preprocessing on the multi-granularity video summary of the agricultural video to be detected to obtain the candidate word sequence {w1, w2, ..., w n}, the candidate word sequence {w1,w2,...,w n Inputting the RoBERTa word vector model yields candidate semantic word vectors {v1, v2, ..., v} for video summarization. n};

[0037] Step 322: Directly obtain the global semantic vector v of the multi-granularity video summary of the agricultural video to be detected by using the RoBERTa-based global semantic vector extraction model. all ;

[0038] Step 323: Calculate the candidate semantic word vectors {v1, v2, ..., v} respectively. n} and the global semantic vector v all Based on the similarity scores, the N keywords with the highest scores are selected as the final extraction result.

[0039] This invention also discloses an agricultural video tag generation device for multi-resolution analysis, implementing the agricultural video tag generation method for multi-resolution analysis as described in any of the above claims, comprising:

[0040] The frame vector acquisition module is used to obtain frame vectors carrying location codes from the agricultural video to be detected;

[0041] The decoding module is used to input the frame vector carrying the position encoding into the multi-layer encoder, extract object features of different granularities from the agricultural video to be detected and the video summary of the agricultural video to be detected to form a multi-granular agricultural video encoding vector, and input the multi-granular agricultural video encoding vector into the multi-layer decoder. The decoding output of each layer of the multi-layer decoder and the encoding output of each layer of the multi-layer encoder are used as the input of the next layer of decoder, and finally the final decoded feature vector after multiple decodings is obtained.

[0042] The tag generation module is used to obtain multiple pending agricultural video summaries and the confidence level of each pending agricultural video summary based on the final decoded feature vector, determine the multi-granularity video summary of the agricultural video to be detected based on the confidence level, and obtain the storage tag of the agricultural video to be detected based on the multi-granularity video summary of the agricultural video to be detected.

[0043] By employing the above-described technical solution, this invention offers the following advantages: it maintains high accuracy and generalization ability even when dealing with complex agricultural videos. Furthermore, this invention reduces resource consumption, increases processing speed, and improves user experience. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments recorded in the embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0045] Figure 1 This is a flowchart illustrating a method for generating agricultural video tags using multi-resolution analysis, according to an embodiment of the present invention.

[0046] Figure 2 This is a partial schematic diagram of a multi-resolution analysis method for generating agricultural video tags according to an embodiment of the present invention. Detailed Implementation

[0047] The present invention will be further described in conjunction with the accompanying drawings and embodiments. The described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art should fall within the protection scope of the present invention.

[0048] See Figure 1 and Figure 2 This invention provides an embodiment of a multi-resolution analysis method for generating agricultural video tags, which includes the following steps:

[0049] S1: Acquire the agricultural video to be tested;

[0050] S2: The acquired agricultural video is segmented into frames, and object features of different granularities are extracted from each frame segment. Each frame segment may contain multiple different object features. Simultaneously, event summary detection is performed on the agricultural video frame segments to be detected, thereby obtaining a video summary for the corresponding frame segments.

[0051] Specifically, the acquired agricultural video is segmented into frames, and object features of different granularities are extracted from each frame segment. Each frame segment may contain multiple different object features. Simultaneously, event summary detection is performed on the agricultural video frame segments to be detected, thereby obtaining a video summary for the corresponding frame segment.

[0052] Since agricultural videos contain a large amount of information, fine-grained object feature extraction is required. This invention mainly focuses on crop features Z in agricultural videos. crop Agricultural meteorological characteristics Z agro Characteristics of agricultural activities Z act and agricultural building features Z pestFurthermore, since directly generating video summaries from videos will produce coarse sentences without fine-grained entities, it is necessary to extract summaries from frame segments. At the same time, supplementing the summary information with the tag information carried by fine-grained object features is beneficial to the generation of the final video summary.

[0053] S2 includes S2.1 and S2.2.

[0054] S2.1: Perform frame segmentation on the acquired agricultural video and extract crop features Z from each frame segment. crop Agricultural meteorological characteristics Z agro Characteristics of agricultural activities Z act and characteristics of agricultural pests Z pest .

[0055] Different feature extraction methods are based on different pre-trained models.

[0056] S2.1 includes S2.1.1 to S2.1.4.

[0057] S2.1.1: Use the vit model pre-trained on the Agriculture crop images dataset to extract crop features for each frame segment. If extracted, obtain the crop feature vector to be encoded for each frame segment. Then, use the pre-built crop feature recognition library (containing common types such as soybean, corn, wheat, and rice) to label the crop feature vector to be encoded for each video segment, thereby obtaining the crop feature vector to be encoded for each video frame segment carrying crop labels.

[0058] S2.1.2: Use the ResNet50 model pre-trained on the Multi-class Weather dataset (MWD) to extract agricultural meteorological features for each frame segment. If extracted, obtain the agricultural meteorological feature vector to be encoded for each frame segment. Then, use the pre-built agricultural meteorological feature recognition library (including common types such as cloudy, rain, and sunshine) to label the agricultural meteorological feature vector to be encoded for each video segment, thereby obtaining the agricultural meteorological feature vector to be encoded for each video frame segment carrying agricultural meteorological labels.

[0059] S2.1.3: Use the vit model pre-trained on the Agricultural-Vision dataset to extract agricultural activity features for each frame segment. If extracted, obtain the agricultural activity feature vector to be encoded for each frame segment. Then, use the pre-built agricultural activity feature recognition library (including common types such as irrigation, planting, and harvesting) to label the agricultural activity feature vector to be encoded for each video segment, thereby obtaining the agricultural activity feature vector to be encoded for each video frame segment carrying agricultural activity labels.

[0060] S2.1.4: Use the YOLOV8 model pre-trained on the IP102 dataset to extract agricultural pest features for each frame segment. If extracted, obtain the agricultural pest feature vector to be encoded for each frame segment. Then, use the pre-built agricultural pest feature recognition library (including common types such as aphids, cotton bollworms, rice planthoppers, and corn borers) to label the agricultural pest feature vector to be encoded for each video segment, thereby obtaining the agricultural pest feature vector to be encoded for each video frame segment carrying agricultural pest labels.

[0061] S2.2: Use the pre-trained PDVC model to perform event summary detection on the agricultural video to be detected frame by frame, and select the summary with the highest confidence as the initial video summary for each frame.

[0062] S3: Use the pre-trained Swin Transformer model to extract frame vectors containing multi-scale information from the agricultural video to be detected frame by frame, and initialize the frame vectors with position encoding to obtain frame vectors carrying position encoding.

[0063] Specifically, a pre-trained Swin Transformer model is used to extract frame vectors Z containing multi-scale information from the agricultural video to be detected, frame by frame. multi The frame vector is initialized with position encoding to obtain a frame vector Z' carrying the position encoding. multi ;

[0064] Since the input model is not a complete image but image patches, and considering that the attention mechanism cannot distinguish the relative positions of features, a positional encoding E is added to the frame vector. pos The calculation formula is as follows:

[0065] Z' multi =Z multi +E pos

[0066] S4: Input the encoded frame vector obtained in S3 into the multi-layer encoder. At the same time, the object features of different granularities obtained in S2 and the obtained video summary are encoded together to form a multi-granularity agricultural video encoded vector. This encoded vector is then input into the multi-layer decoder. The decoding output of each layer decoder and the encoding output of each layer encoder are used as the input of the next layer decoder to obtain the final decoded feature vector after multiple decodings.

[0067] S4 includes S4.1 to S4.5.

[0068] S4.1: The encoded frame vector Z' obtained in S3 multiThe input is fed into a multi-layer Transformer encoder. Since the Transformer encoder does not change the dimension of the input features, the output has the same dimension as the input, and the encoder can be stacked at different depths.

[0069] S4.2: Encode the possible crop feature vectors, agricultural meteorological feature vectors, agricultural activity feature vectors, and agricultural pest feature vectors that need to be encoded in each frame segment into word vectors using word2vec, and encode the initial video summary of each frame segment into sentence vectors with the same length as the word vectors using sentence encoding;

[0070] S4.3: Combine the sentence vectors and word vectors obtained in S4.2 to form the encoding vector;

[0071] S4.4: Input the encoded vector obtained in S4.3 into the multi-layer Transformer decoder, and at the same time, input the encoded output of the Transformer encoder obtained in S4.1 as the supervision target into the current layer Transformer decoder to obtain the decoding output of the current layer Transformer decoder.

[0072] S4.5: Input the output of the current layer's Transformer decoder into the next layer's Transformer decoder, and simultaneously input the encoding output of the next layer's Transformer encoder as the supervision target into the current layer's Transformer decoder. Finally, obtain the output of the current layer's Transformer decoder. The encoder-decoder layer number N is set to 5. After N iterations, the final decoding output is obtained.

[0073] In one embodiment, the encoded frame vector Z' obtained in S3 multi The input is fed into a multilayer encoder to obtain the encoded vector Z. _E Simultaneously, the object features of different granularities obtained in S2 are combined with the obtained video summary to form a multi-granularity agricultural video coding vector V. multi The encoded vector is then input into a multi-layer decoder, and the decoding output Z of each layer decoder is... _Di The encoding output Z of each encoder layer _Ei As input to the next layer of decoder, the final decoded feature vector V is obtained after multiple decodings. feature ;

[0074]

[0075] Among them, Z _E1 Z represents the encoded output of the first-layer encoder. _Ei Z represents the encoded output of the i-th layer encoder._D1 Z represents the decoding output of the first-layer decoder. _Di This represents the decoding output of the i-th layer decoder. This is the decoding output of the Nth layer decoder, where N is the number of layers in the decoder, Transformer_Encoder is the encoder, and Transformer_Decoder is the decoder.

[0076] S5: Input the final decoded feature vector obtained in S4 into the confidence feedforward model and the LSTM model with attention mechanism, respectively, to obtain multiple undetermined agricultural video summaries output by the LSTM model and the confidence level of each undetermined agricultural video summary output by the confidence feedforward model. Finally, select the undetermined video summary with the highest confidence level as the multi-granularity video summary of the agricultural video to be detected.

[0077] S6: Extract multi-granularity video summary keywords from the agricultural videos to be detected using a word vector extraction model based on the RoBERTa pre-trained model and a global semantic vector extraction model. Use the extracted keywords as storage tags for the agricultural videos to be detected.

[0078] S6 includes S6.1 to S6.3.

[0079] S6.1: Perform text preprocessing on the multi-granularity video summary of the agricultural video to be detected obtained in S5, including word segmentation, stop word processing, etc., to obtain the candidate word sequence {w1, w2, ..., w n}, the candidate word sequence {w1,w2,...,w n Inputting the RoBERTa word vector model yields its candidate semantic word vectors {v1, v2, ..., v}. n};

[0080] S6.2: The multi-granularity video summary of the agricultural video to be detected obtained in S5 is directly processed by the RoBERTa-based global semantic vector extraction model to obtain the global semantic vector v of the summary text. all ;

[0081] S6.3: Calculate candidate semantic word vectors {v1, v2, ..., v...} based on cosine similarity. n} and the global semantic vector v all Based on the similarity scores, the N keywords with the highest scores are selected as the final extraction result.

[0082] Where v i With v all The formula for calculating similarity is:

[0083]

[0084] Wherein, S(vi ,v all ) for v i With v all similarity, v i Let v be the i-th candidate semantic word vector. i_j For v i The j-th dimension, v all_j For v all The j-th dimension, d is the vector dimension, which can be set to 768 dimensions.

[0085] The present invention also provides an embodiment of an agricultural video tag generation device for multi-resolution analysis, which implements the agricultural video tag generation method for multi-resolution analysis described in the above embodiments, and includes:

[0086] The frame vector acquisition module is used to obtain frame vectors carrying location codes from the agricultural video to be detected;

[0087] The decoding module is used to input the frame vector carrying the position encoding into the multi-layer encoder, extract object features of different granularities from the agricultural video to be detected and the video summary of the agricultural video to be detected to form a multi-granular agricultural video encoding vector, and input the multi-granular agricultural video encoding vector into the multi-layer decoder. The decoding output of each layer of the multi-layer decoder and the encoding output of each layer of the multi-layer encoder are used as the input of the next layer of decoder, and finally the final decoded feature vector after multiple decodings is obtained.

[0088] The tag generation module is used to obtain multiple pending agricultural video summaries and the confidence level of each pending agricultural video summary based on the final decoded feature vector, determine the multi-granularity video summary of the agricultural video to be detected based on the confidence level, and obtain the storage tag of the agricultural video to be detected based on the multi-granularity video summary of the agricultural video to be detected.

[0089] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for generating agricultural video tags using multi-resolution analysis, characterized in that, include: Step 1: Obtain the frame vector carrying the location code from the agricultural video to be detected; Step 2: Input the frame vector carrying the position encoding into the multi-layer encoder. The object features of different granularities extracted from the agricultural video to be detected and the video summary of the agricultural video to be detected constitute a multi-granularity agricultural video encoding vector. The multi-granularity agricultural video encoding vector is then input into the multi-layer decoder. The decoding output of each layer of the multi-layer decoder and the encoding output of each layer of the multi-layer encoder are used together as the input of the next layer of the decoder. Finally, the final decoded feature vector after multiple decodings is obtained. Step 3: Based on the final decoded feature vector, obtain multiple pending agricultural video summaries and the confidence level corresponding to each pending agricultural video summary. Determine the multi-granularity video summaries of the agricultural videos to be detected based on the confidence levels. Based on the multi-granularity video summaries of the agricultural videos to be detected, obtain the storage tags of the agricultural videos to be detected.

2. The method for generating agricultural video tags using multi-resolution analysis according to claim 1, characterized in that, Step 1 includes: Step 11: Perform frame segmentation on the agricultural video to be detected, extract object features of different granularities from the frame segments, where each frame segment may contain multiple different object features; perform event summary detection on the frame segments of the agricultural video to be detected, and at the same time, use the tag information carried by the object features to complete the detected summary information, and finally obtain the video summary of the corresponding frame segment. Step 12: Extract frame vectors containing multi-scale information frame by frame from the agricultural video to be detected, and initialize the frame vectors with position encoding to obtain frame vectors carrying position encoding.

3. The method for generating agricultural video tags using multi-resolution analysis according to claim 2, characterized in that, Step 11 includes: Step 11: Extract crop features, agricultural meteorological features, agricultural activity features, and agricultural pest features from the agricultural video clips to be detected; Step 12: Use the pre-trained PDVC model to perform event summary detection on the agricultural video to be detected frame by frame, and select the summary with the highest confidence as the initial video summary for each frame segment; Step 13: Complete the initial video summary with the tag information carried by the object features to finally obtain the video summary of the corresponding frame segment.

4. The method for generating agricultural video tags using multi-resolution analysis according to claim 3, characterized in that, Step 11 includes: Step 111: Use the vit model pre-trained on the Agriculture crop images dataset to extract crop features for each agricultural video frame segment to be detected. If crop features are extracted, all extracted crop features constitute the crop feature vector to be encoded for each agricultural video frame segment to be detected. Then, use the pre-built crop feature recognition library to label the crop feature vector to be encoded for each agricultural video frame segment to be detected, thereby obtaining the crop feature vector to be encoded for each agricultural video frame segment to be detected carrying crop labels. Step 112: Use the ResNet50 model pre-trained on the Multi-class weather dataset to extract agricultural meteorological features for each agricultural video frame segment to be detected. If agricultural meteorological features are extracted, all extracted agricultural meteorological features constitute the agricultural meteorological feature vector to be encoded for each agricultural video frame segment to be detected. Then, use the pre-built agricultural meteorological feature recognition library to label the agricultural meteorological feature vector to be encoded for each agricultural video frame segment to be detected, thereby obtaining the agricultural meteorological feature vector to be encoded for each agricultural video frame segment to be detected carrying agricultural meteorological labels. Step 113: Use the vit model pre-trained on the Agricultural-Vision dataset to extract agricultural activity features for each agricultural video frame segment to be detected. If agricultural activity features are extracted, all extracted agricultural activity features constitute the agricultural activity feature vector to be encoded for each agricultural video frame segment to be detected. Then, use the pre-built agricultural activity feature recognition library to label the agricultural activity feature vector to be encoded for each agricultural video frame segment to be detected, thereby obtaining the agricultural activity feature vector to be encoded for each agricultural video frame segment to be detected carrying agricultural activity labels. Step 114: Use the YOLO V8 model pre-trained on the IP102 dataset to extract agricultural pest features for each agricultural video frame segment to be detected. If agricultural pest features are extracted, all extracted agricultural pest features constitute the agricultural pest feature vector to be encoded for each agricultural video frame segment to be detected. Then, use the pre-built agricultural pest feature recognition library to label the agricultural pest feature vector to be encoded for each agricultural video frame segment to be detected, thereby obtaining the agricultural pest feature vector to be encoded for each agricultural video frame segment to be detected carrying agricultural pest labels.

5. The method for generating agricultural video tags using multi-resolution analysis according to claim 1, characterized in that, Step 2 includes: Step 21: Convert the frame vector Z' carrying the position code obtained in Step 1 into a frame vector Z'. multi Input to the multi-layer Transformer encoder; Step 22: Encode the possible crop feature vectors, agricultural meteorological feature vectors, agricultural activity feature vectors, and agricultural pest feature vectors that may exist in each agricultural video frame segment to be detected into word vectors, and encode the initial video summary of each agricultural video frame segment to be detected into sentence vectors with the same length as the word vectors through sentence encoding; Step 23: Combine the sentence vector and word vector obtained in Step 22 to form an encoding vector; Step 24: Input the encoded vector obtained in step 23 into the multi-layer Transformer decoder, and at the same time, input the encoded output of the Transformer encoder obtained in step 21 as the supervision target into the current layer Transformer decoder to obtain the decoding output of the current layer Transformer decoder. Step 25: Input the output of the current layer's Transformer decoder into the next layer's Transformer decoder, and simultaneously input the encoding output of the next layer's Transformer encoder as the supervision target into the current layer's Transformer decoder. Finally, obtain the output of the current layer's Transformer decoder. After N iterations, obtain the final decoding output; N is the number of encoder-decoder layers.

6. The method for generating agricultural video tags using multi-resolution analysis according to claim 1, characterized in that, Step 3 includes: Step 31: Based on the final decoded feature vector, obtain multiple pending agricultural video summaries and the confidence level of each pending agricultural video summary. Finally, determine the pending agricultural video summary with the highest confidence level as the multi-granularity video summary of the agricultural video to be detected. Step 32: Extract keywords from the multi-granularity video summary of the agricultural video to be detected, and use the extracted keywords as storage tags for the agricultural video to be detected.

7. The method for generating agricultural video tags using multi-resolution analysis according to claim 6, characterized in that, Step 31 includes: The final decoded feature vectors are input into the confidence feedforward model and the LSTM model with attention mechanism, respectively, to obtain multiple undetermined agricultural video summaries output by the LSTM model and the confidence level of each undetermined agricultural video summary output by the confidence feedforward model. Finally, the undetermined video summary with the highest confidence level is selected as the multi-granularity video summary of the agricultural video to be detected.

8. The method for generating agricultural video tags using multi-resolution analysis according to claim 7, characterized in that, Step 32 includes: Multi-granular video summary keywords were extracted from agricultural videos to be detected using a RoBERTa-based word vector and global semantic vector extraction model, and the extracted keywords were used as storage tags for the agricultural videos to be detected.

9. The method for generating agricultural video tags using multi-resolution analysis according to claim 6 or 8, characterized in that, Step 32 specifically includes: Step 321: Perform text preprocessing on the multi-granularity video summary of the agricultural video to be detected to obtain the candidate word sequence {w1, w2, ..., w n }, the candidate word sequence {w1,w2,...,w n Inputting the RoBERTa word vector model yields candidate semantic word vectors {v1, v2, ..., v} for video summarization. n }; Step 322: Directly obtain the global semantic vector v of the multi-granularity video summary of the agricultural video to be detected by using the RoBERTa-based global semantic vector extraction model. all ; Step 323: Calculate the candidate semantic word vectors {v1, v2, ..., v} respectively. n } and global semantic vector v all Based on the similarity scores, the N keywords with the highest scores are selected as the final extraction result.

10. An agricultural video tag generation device for multi-resolution analysis, implementing the agricultural video tag generation method for multi-resolution analysis as described in any one of claims 1-9, characterized in that, include: The frame vector acquisition module is used to obtain frame vectors carrying location codes from the agricultural video to be detected; The decoding module is used to input the frame vector carrying the position encoding into the multi-layer encoder, extract object features of different granularities from the agricultural video to be detected and the video summary of the agricultural video to be detected to form a multi-granular agricultural video encoding vector, and input the multi-granular agricultural video encoding vector into the multi-layer decoder. The decoding output of each layer of the multi-layer decoder and the encoding output of each layer of the multi-layer encoder are used as the input of the next layer of decoder, and finally the final decoded feature vector after multiple decodings is obtained. The tag generation module is used to obtain multiple pending agricultural video summaries and the confidence level of each pending agricultural video summary based on the final decoded feature vector, determine the multi-granularity video summary of the agricultural video to be detected based on the confidence level, and obtain the storage tag of the agricultural video to be detected based on the multi-granularity video summary of the agricultural video to be detected.

Images