Multi-resolution analysis agricultural video tag generation method and device

Through the combination of multi-resolution analysis and multi-layer Transformer encoder decoder, the multi-grained features and abstracts of agricultural videos are extracted, and the accuracy and efficiency of complex video tag generation in the existing technology is solved, achieving efficient and accurate agriculture Video tag generation.

CN119048956AActive Publication Date: 2024-11-29XIDIAN UNIV
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202411134727.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-19
Publication Date
2024-11-29
Estimated Expiration
2044-08-19

AI Technical Summary

Technical Problem

The existing agricultural video tag generation technology has limited accuracy and generalization capabilities when dealing with complex videos, high resource consumption and slow processing speed, which affects the user experience.

Method used

Multi-resolution analysis method is used to extract object features and video abstracts of different granularities through multi-layer encoder and decoder, and determine the multi-grained video abstracts with confidence to generate efficient agricultural video tags. The specific steps include frame segmentation, feature extraction, location encoding, multi-layer Transformer encoding and decoding, and finally extracting keywords using the RoBERTa model.

Benefits of technology

It improves the accuracy and generalization ability of tag generation of complex agricultural videos, reduces resource consumption, and improves processing speed and user experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119048956A_ABST
    Figure CN119048956A_ABST
Patent Text Reader

Abstract

The invention discloses a multi-resolution analysis agricultural video tag generation method and device. The method comprises the following steps: obtaining a frame vector carrying a position code from a to-be-detected agricultural video; inputting a frame vector carrying a position code into a multi-layer encoder, inputting a multi-granularity agricultural video coding vector into a multi-layer decoder, and taking a decoding output result of each layer of decoder and a coding output result of each layer of encoder in the multi-layer encoder as input of a next layer of decoder, and finally, obtaining a final decoding feature vector after multiple times of decoding. And obtaining a plurality of to-be-determined agricultural video abstracts and confidence coefficients corresponding to the to-be-determined agricultural video abstracts according to the final decoding feature vectors, determining a multi-granularity video abstract of the to-be-detected agricultural video according to the confidence coefficients, and obtaining a storage tag of the to-be-detected agricultural video based on the multi-granularity video abstract of the to-be-detected agricultural video. The method still has very high accuracy and generalization ability when being used for coping with complex agricultural videos.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of video processing, and in particular to a method and device for generating agricultural video labels for multi-resolution analysis. Background Art

[0002] In the development of data agriculture, video has become an important carrier for agricultural information dissemination, and labels are key tools for obtaining accurate information and positioning data in multiple fields, playing an increasingly important role in crop growth monitoring, pest control, and evaluation of key behaviors in agricultural production. However, manual analysis of these video data is time-consuming and labor-intensive, and it is difficult to cope with the growing amount of data. Therefore, opening up an efficient and automated video label generation technology is of great significance to improving the utilization efficiency and management level of agricultural video data.

[0003] Video tagging is a tool that can accurately describe the content of a video and plays an important role in the organization and retrieval of agricultural videos. By analyzing multi-dimensional elements such as images, text, dubbing, and behavior in agricultural videos, video tags provide users with a comprehensive description of the video content. They are usually used in scenarios such as agricultural knowledge graph construction, production process optimization, and crop growth monitoring.

[0004] However, current agricultural video label generation technology has certain limitations. Specifically, although the traditional single-resolution video analysis method can generate accurate labels in some cases, its accuracy and generalization ability are limited when dealing with complex agricultural videos. In addition, existing technologies also have shortcomings in resource consumption, processing speed and user experience. Summary of the invention

[0005] In order to solve the above problems existing in the prior art, the present invention provides a method and device for generating agricultural video labels by multi-resolution analysis.

[0006] The present invention discloses a method for generating agricultural video labels by multi-resolution analysis, which comprises:

[0007] Step 1: Obtain a frame vector carrying position coding from the agricultural video to be detected;

[0008] Step 2: Input the frame vector carrying the position code into the multi-layer encoder, and form a multi-granularity agricultural video coding vector with the object features of different granularities extracted from the agricultural video to be detected and the video summary of the agricultural video to be detected, and input the multi-granularity agricultural video coding vector into the multi-layer decoder. The decoding output result of each layer of the decoder in the multi-layer decoder and the encoding output result of each layer of the encoder in the multi-layer encoder are used as the input of the next layer of decoder, and finally the final decoding feature vector after multiple decoding is obtained;

[0009] Step 3: According to the final decoded feature vector, multiple pending agricultural video summaries and the confidence corresponding to each pending agricultural video summary are obtained, and the multi-granularity video summary of the agricultural video to be detected is determined according to the confidence. Based on the multi-granularity video summary of the agricultural video to be detected, the storage label of the agricultural video to be detected is obtained.

[0010] Furthermore, the step 1 comprises:

[0011] Step 11: Divide the agricultural video to be detected into frames and extract object features of different granularities on the frame segments, where each frame segment may contain multiple different object features; perform event summary detection on the frame segments of the agricultural video to be detected, and at the same time, use the label information carried by the object features to complete the summary information obtained by the detection, and finally obtain the video summary of the corresponding frame segment.

[0012] Step 12: Extract frame vectors containing multi-scale information from the agricultural video to be detected frame by frame, and initialize the position encoding of the frame vectors to obtain frame vectors carrying the position encoding.

[0013] Furthermore, the step 11 comprises:

[0014] Step 11: extracting crop features, agricultural meteorological features, agricultural activity features, and agricultural pest features from the agricultural video clips to be detected;

[0015] Step 12: Use the pre-trained PDVC model to perform event summary detection on the agricultural video to be detected according to the frame segments, and select the summary with the highest confidence as the initial video summary of each frame segment;

[0016] Step 13: The label information carried by the object features is used to complete the initial video summary, and finally a video summary of the corresponding frame segment is obtained.

[0017] Furthermore, the step 11 comprises:

[0018] Step 111: using the VIT model pre-trained on the Agriculture crop images dataset to extract crop features from each agricultural video frame segment to be detected. If crop features are extracted, all the extracted crop features constitute the crop feature vector to be encoded of each agricultural video frame segment to be detected, and using the constructed crop feature recognition library to label the crop feature vector to be encoded of each agricultural video frame segment to be detected, so as to obtain the crop feature vector to be encoded carrying the crop label of each agricultural video frame segment to be detected;

[0019] Step 112: using the resnet50 model pre-trained on the Multi-class weather dataset to extract agricultural meteorological features from each agricultural video frame segment to be detected. If agricultural meteorological features are extracted, all the extracted agricultural meteorological features constitute the agricultural meteorological feature vector to be encoded for each agricultural video frame segment to be detected, and using the constructed agricultural meteorological feature recognition library to label the agricultural meteorological feature vector to be encoded for each agricultural video frame segment to be detected, so as to obtain the agricultural meteorological feature vector to be encoded for each agricultural video frame segment to be detected carrying the agricultural meteorological label;

[0020] Step 113: using the vit model pre-trained on the Agricultural-Vision dataset to extract agricultural activity features from each agricultural video frame segment to be detected. If agricultural activity features are extracted, all the extracted agricultural activity features constitute the agricultural activity feature vector to be encoded of each agricultural video frame segment to be detected, and using the constructed agricultural activity feature recognition library to label the agricultural activity feature vector to be encoded of each agricultural video frame segment to be detected, so as to obtain the agricultural activity feature vector to be encoded carrying the agricultural activity label for each agricultural video frame segment to be detected;

[0021] Step 114: Use the YOLO V8 model pre-trained on the IP102 dataset to extract agricultural pest features from each agricultural video frame segment to be detected. If agricultural pest features are extracted, all extracted agricultural pest features constitute the agricultural pest feature vector to be encoded for each agricultural video frame segment to be detected, and use the constructed agricultural pest feature recognition library to label the agricultural pest feature vector to be encoded for each agricultural video frame segment to be detected, so as to obtain the agricultural pest feature vector to be encoded for each agricultural video frame segment to be detected carrying the agricultural pest label.

[0022] Furthermore, the step 2 comprises:

[0023] Step 21: The frame vector Z' carrying the position code obtained in step 1 mu lti is input into the multi-layer Transformer encoder;

[0024] Step 22: Encode the crop feature vectors to be encoded, the agricultural meteorological feature vectors to be encoded, the agricultural activity feature vectors to be encoded, and the agricultural pest feature vectors to be encoded that may exist in each agricultural video frame segment to be detected into word vectors, and form a sentence vector with the same length as the word vector through sentence encoding of the initial video summary of each agricultural video frame segment to be detected;

[0025] Step 23: Combine the sentence vector and word vector obtained in step 22 into an encoding vector;

[0026] Step 24: The encoding vector obtained in step 23 is fed into the multi-layer Transformer decoder, and the encoding output result of the Transformer encoder obtained in step 21 is input into the current layer Transformer decoder as the supervision target to obtain the decoding output result of the current layer Transformer decoder;

[0027] Step 25: Input the output result of the Transformer decoder of the current layer into the Transformer decoder of the next layer, and at the same time input the encoded output result of the Transformer encoder of the next layer as the supervision target into the Transformer decoder of this layer, and finally obtain the output result of the Transformer decoder of this layer. After N iterations, the final decoding output result is obtained; N is the number of encoder-decoder layers.

[0028] Furthermore, the step 3 comprises:

[0029] Step 31: According to the final decoded feature vector, a plurality of pending agricultural video summaries and the confidence corresponding to each pending agricultural video summary are obtained, and finally the pending agricultural video summary with the highest confidence is determined as the multi-granularity video summary of the agricultural video to be detected;

[0030] Step 32: extract keywords from the multi-granularity video summary of the agricultural video to be detected, and use the extracted keywords as storage tags of the agricultural video to be detected.

[0031] Furthermore, the step 31 comprises:

[0032] The final decoded feature vector is respectively input into the confidence feedforward model and the LSTM model with attention mechanism to obtain multiple pending agricultural video summaries output by the LSTM model and the confidence corresponding to each pending agricultural video summary output by the confidence feedforward model. Finally, the pending video summary with the highest confidence is selected as the multi-granularity video summary of the agricultural video to be detected.

[0033] Further, the step 32 comprises:

[0034] Multi-granularity video summary keywords of the agricultural video to be detected are extracted through the RoBERTa-based word embedding and global semantic vector extraction model, and the extracted keywords are used as storage labels of the agricultural video to be detected.

[0035] Furthermore, the step 32 specifically includes:

[0036] Step 321: Perform text preprocessing on the multi-granularity video summary of the agricultural video to be detected to obtain a candidate word sequence {w1, w2, ..., w n}, the candidate word sequence {w1,w2,...,w n} Input the RoBERTa word vector model to obtain the candidate semantic word vectors of the video summary {v1,v2,...,v n};

[0037] Step 322: directly obtain the global semantic vector v of the multi-granularity video summary of the agricultural video to be detected through the global semantic vector extraction model based on RoBERTa all ;

[0038] Step 323: Calculate the candidate semantic word vectors {v1, v2, ..., v n} and the global semantic vector v all The similarity of the two keywords is calculated and the N keywords with the highest scores are selected as the final results of the extraction.

[0039] The present invention also discloses a device for generating agricultural video labels for multi-resolution analysis, which implements any of the above-mentioned methods for generating agricultural video labels for multi-resolution analysis, and comprises:

[0040] A frame vector acquisition module is used to obtain a frame vector carrying a position code from the agricultural video to be detected;

[0041] A decoding module is used to input the frame vector carrying the position code into a multi-layer encoder, to form a multi-granularity agricultural video coding vector with object features of different granularities extracted from the agricultural video to be detected and the video summary of the agricultural video to be detected, and to input the multi-granularity agricultural video coding vector into a multi-layer decoder, and the decoding output result of each layer of the decoder in the multi-layer decoder and the encoding output result of each layer of the encoder in the multi-layer encoder are used together as the input of the next layer of the decoder, and finally a final decoding feature vector after multiple decoding is obtained;

[0042] The label generation module is used to obtain multiple pending agricultural video summaries and the confidence corresponding to each pending agricultural video summary based on the final decoded feature vector, determine the multi-granularity video summary of the agricultural video to be detected based on the confidence, and obtain the storage label of the agricultural video to be detected based on the multi-granularity video summary of the agricultural video to be detected.

[0043] Due to the adoption of the above technical solution, the present invention has the following advantages: when dealing with complex agricultural videos, it still has high accuracy and generalization ability. In addition, the present invention reduces resource consumption, increases processing speed, and improves user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0045] Figure 1 A schematic diagram of a flow chart of a method for generating agricultural video labels by multi-resolution analysis according to an embodiment of the present invention;

[0046] Figure 2 A schematic diagram of a partial process of a method for generating agricultural video labels through multi-resolution analysis according to an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The present invention is further described in conjunction with the accompanying drawings and embodiments, and the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field should fall within the scope of protection of the embodiments of the present invention.

[0048] See also Figure 1 and Figure 2 The present invention provides an embodiment of a method for generating agricultural video labels by multi-resolution analysis, which comprises the following steps:

[0049] S1: Obtain the agricultural video to be detected;

[0050] S2: The acquired agricultural video is processed into frames, and object features of different granularities are extracted from the frame segments, where each frame segment may contain multiple different object features. While extracting different object features, event summary detection is performed on the agricultural video frame segments to be detected, so as to obtain the video summary of the corresponding frame segments.

[0051] Specifically, the acquired agricultural video is processed into frames to obtain frame segments, and object features of different granularities are extracted from each frame segment, wherein each frame segment may contain multiple different object features. While extracting different object features, event summary detection is performed on the agricultural video frame segments to be detected, thereby obtaining the video summary of the corresponding frame segments.

[0052] Among them, since agricultural videos contain a lot of information, it is necessary to extract fine-grained object features. This paper mainly focuses on the crop features Z in agricultural videos. crop 、Agrometeorological characteristics agro , Characteristics of agricultural activities Z act and agricultural building feature Z pest, and since generating video summaries directly in videos will produce rough sentences without fine-grained entities, it is necessary to extract summaries from frame fragments, and at the same time, the summary information is completed by the label information carried by the fine-grained object features, which is conducive to the generation of the final video summary.

[0053] S2 includes S2.1 and S2.2.

[0054] S2.1: Frame the acquired agricultural video and extract crop features Z in each frame segment crop 、Agrometeorological characteristics agro , Characteristics of agricultural activities Z act and agricultural pest characteristics Z pest .

[0055] Among them, different feature extractions are based on different pre-training models.

[0056] S2.1 includes S2.1.1 to S2.1.4.

[0057] S2.1.1: Use the vit model pre-trained on the Agriculture crop images dataset to extract crop features from each frame segment. If extracted, obtain the feature vector of the crop to be encoded for each frame segment, and use the constructed crop feature recognition library (including common types such as soybeans, corn, wheat, and rice) to label the feature vector of the crop to be encoded for each video segment, so as to obtain the feature vector of the crop to be encoded with the crop label for each video frame segment;

[0058] S2.1.2: Use the resnet50 model pre-trained on the Multi-class weather dataset (MWD) dataset to extract agricultural meteorological features for each frame segment. If extracted, obtain the agricultural meteorological feature vector to be encoded for each frame segment, and use the constructed agricultural meteorological feature recognition library (including common types such as cloudy, rainy, sunny, etc.) to annotate the agricultural meteorological feature vector to be encoded for each video segment, so as to obtain the agricultural meteorological feature vector to be encoded for each video frame segment with agricultural meteorological labels;

[0059] S2.1.3: Use the vit model pre-trained on the Agricultural-Vision dataset to extract agricultural activity features from each frame segment. If extracted, obtain the feature vector of the agricultural activity to be encoded for each frame segment, and use the constructed agricultural activity feature recognition library (including common types such as irrigation, planting, and harvesting) to label the feature vector of the agricultural activity to be encoded for each video segment, so as to obtain the feature vector of the agricultural activity to be encoded for each video frame segment carrying the agricultural activity label;

[0060] S2.1.4: Use the YOLOV8 model pre-trained on the IP102 dataset to extract agricultural pest features from each frame segment. If extracted, obtain the feature vector of agricultural pests to be encoded for each frame segment, and use the constructed agricultural pest feature recognition library (including common types such as aphids, cotton bollworms, rice planthoppers, and corn borers) to label the feature vector of agricultural pests to be encoded for each video segment, so as to obtain the feature vector of agricultural pests to be encoded with agricultural pest labels for each video frame segment;

[0061] S2.2: Use the pre-trained PDVC model to perform event summary detection on the agricultural video according to the frame segments, and select the summary with the highest confidence as the initial video summary of each frame segment.

[0062] S3: Use the pre-trained Swin Transformer model to extract frame vectors containing multi-scale information from the agricultural video to be detected frame by frame, and initialize the position encoding of the frame vector to obtain a frame vector carrying the position encoding;

[0063] Specifically, the pre-trained Swin Transformer model is used to extract frame vectors Z containing multi-scale information from the agricultural video to be detected frame by frame. multi , and initialize the position encoding of the frame vector to obtain the frame vector Z' carrying the position encoding multi ;

[0064] Among them, since the input model is not a complete image but image patches, considering that the attention mechanism cannot distinguish the relative position of the features, the position encoding E is added to the frame vector pos The calculation formula is as follows:

[0065] Z' multi =Z multi +E pos

[0066] S4: Input the coded frame vector obtained in S3 into the multi-layer encoder, and simultaneously encode the object features of different granularities obtained in S2 and the obtained video summary to form a multi-granularity agricultural video coded vector, and input the coded vector into the multi-layer decoder. The decoding output result of each layer of decoder and the encoding output result of each layer of encoder are used as the input of the next layer of decoder, and the final decoded feature vector after multiple decoding is obtained;

[0067] S4 includes S4.1 to S4.5.

[0068] S4.1: The coded frame vector Z' obtained in S3 multiInput to the multi-layer Transformer encoder. Since the Transformer encoder does not change the dimension of the input features, the output has the same dimension as the input and the encoder can be deeply stacked.

[0069] S4.2: Encode the crop feature vectors to be encoded, the agricultural meteorological feature vectors to be encoded, the agricultural activity feature vectors to be encoded, and the agricultural pest feature vectors to be encoded that may exist in each frame segment into word vectors through word2vec, and form a sentence vector with the same length as the word vector through sentence encoding of the initial video summary of each frame segment;

[0070] S4.3: Combine the sentence vector and word vector obtained in S4.2 into an encoding vector;

[0071] S4.4: The encoding vector obtained in S4.3 is sent to the multi-layer Transformer decoder, and the encoding output result of the Transformer encoder obtained in S4.1 is input as the supervision target into the current layer Transformer decoder to obtain the decoding output result of the current layer Transformer decoder;

[0072] S4.5: The output result of the Transformer decoder of the current layer is input into the Transformer decoder of the next layer. At the same time, the encoded output result of the Transformer encoder of the next layer is input into the Transformer decoder of this layer as the supervision target. Finally, the output result of the Transformer decoder of this layer is obtained. The number of encoder-decoder layers N is set to 5. After N iterations, the final decoding output result is obtained.

[0073] In one embodiment, the coded frame vector Z' obtained in S3 is multi Input into the multi-layer encoder to obtain the encoding vector Z _E At the same time, the object features of different granularities obtained in S2 and the obtained video summary are encoded together to form a multi-granularity agricultural video coding vector V multi , and input the encoded vector into the multi-layer decoder. The decoding output of each layer of decoder is Z _Di The encoding output Z of each layer encoder _Ei As the input of the next layer decoder, the final decoded feature vector V is obtained after multiple decoding feature ;

[0074]

[0075] Among them, Z _E1 is the encoding output result of the first layer encoder, Z _Ei is the encoding output result of the i-th layer encoder, Z_D1 is the decoding output result of the first layer decoder, Z _Di is the decoding output result of the i-th layer decoder, is the decoding output result of the Nth layer decoder, N is the number of decoder layers, Transformer_Encoder is the encoder, and Transformer_Decoder is the decoder.

[0076] S5: The final decoded feature vector obtained in S4 is input into the confidence feedforward model and the LSTM model with attention mechanism respectively, so as to obtain multiple pending agricultural video summaries output by the LSTM model and the confidence corresponding to each pending agricultural video summary output by the confidence feedforward model, and finally the pending video summary with the highest confidence is selected to be determined as the multi-granularity video summary of the agricultural video to be detected;

[0077] S6: The multi-granularity video summary keywords of the agricultural video to be detected are extracted through the word embedding based on the RoBERTa pre-trained model and the global semantic vector extraction model, and the extracted keywords are used as the storage labels of the agricultural video to be detected.

[0078] S6 includes S6.1 to S6.3.

[0079] S6.1: Perform text preprocessing on the multi-granular video summary of the agricultural video to be detected obtained in S5, including word segmentation, stop word processing and other operations, to obtain the candidate word sequence {w1,w2,...,w n}, the candidate word sequence {w1,w2,...,w n} Input the RoBERTa word vector model to obtain its candidate semantic word vectors {v1,v2,...,v n};

[0080] S6.2: The multi-granular video summary of the agricultural video to be detected obtained in S5 is directly applied to the global semantic vector extraction model based on RoBERTa to obtain the global semantic vector v of the summary text. all ;

[0081] S6.3: Calculate the candidate semantic word vectors {v1,v2,...,v n} and the global semantic vector v all The similarity of the two keywords is calculated and the N keywords with the highest scores are selected as the final results of the extraction.

[0082] where v i With v all The similarity calculation formula is:

[0083]

[0084] Among them, S(vi ,v all ) is v i With v all The similarity of i is the i-th candidate semantic word vector, v i_j v i The jth dimension of all_j v all The j-th dimension of , d is the vector dimension, which can be set to 768 dimensions.

[0085] The present invention also provides an embodiment of a device for generating agricultural video labels by multi-resolution analysis, which implements the method for generating agricultural video labels by multi-resolution analysis described in the above embodiment, and comprises:

[0086] A frame vector acquisition module is used to obtain a frame vector carrying a position code from the agricultural video to be detected;

[0087] A decoding module is used to input the frame vector carrying the position code into a multi-layer encoder, to form a multi-granularity agricultural video coding vector with object features of different granularities extracted from the agricultural video to be detected and the video summary of the agricultural video to be detected, and to input the multi-granularity agricultural video coding vector into a multi-layer decoder, and the decoding output result of each layer of the decoder in the multi-layer decoder and the encoding output result of each layer of the encoder in the multi-layer encoder are used together as the input of the next layer of the decoder, and finally a final decoding feature vector after multiple decoding is obtained;

[0088] The label generation module is used to obtain multiple pending agricultural video summaries and the confidence corresponding to each pending agricultural video summary based on the final decoded feature vector, determine the multi-granularity video summary of the agricultural video to be detected based on the confidence, and obtain the storage label of the agricultural video to be detected based on the multi-granularity video summary of the agricultural video to be detected.

[0089] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for generating agricultural video labels based on multi-resolution analysis. It is characterized in that include: Step 1: Obtain a frame vector carrying position coding from the agricultural video to be detected; Step 2: Input the frame vector carrying the position code into the multi-layer encoder, and form a multi-granularity agricultural video coding vector with the object features of different granularities extracted from the agricultural video to be detected and the video summary of the agricultural video to be detected, and input the multi-granularity agricultural video coding vector into the multi-layer decoder. The decoding output result of each layer of the decoder in the multi-layer decoder and the encoding output result of each layer of the encoder in the multi-layer encoder are used as the input of the next layer of decoder, and finally the final decoding feature vector after multiple decoding is obtained; Step 3: According to the final decoded feature vector, multiple pending agricultural video summaries and the confidence corresponding to each pending agricultural video summary are obtained, and the multi-granularity video summary of the agricultural video to be detected is determined according to the confidence. Based on the multi-granularity video summary of the agricultural video to be detected, the storage label of the agricultural video to be detected is obtained.

2. The agricultural video label generation method of multi-resolution analysis according to claim 1, It is characterized in that The step 1 comprises: Step 11: The agricultural video to be detected is processed by dividing the frames into frames, and object features of different granularities are extracted from the frame segments, where each frame segment may contain multiple different object features; event summary detection is performed on the frame segments of the agricultural video to be detected, and the label information carried by the object features is used to complete the summary information obtained by the detection, and finally a video summary of the corresponding frame segment is obtained; Step 12: Extract frame vectors containing multi-scale information from the agricultural video to be detected frame by frame, and initialize the position encoding of the frame vectors to obtain frame vectors carrying the position encoding.

3. The agricultural video label generation method of multi-resolution analysis according to claim 2, It is characterized in that The step 11 comprises: Step 11: extracting crop features, agricultural meteorological features, agricultural activity features, and agricultural pest features from the agricultural video clips to be detected; Step 12: Use the pre-trained PDVC model to perform event summary detection on the agricultural video to be detected according to the frame segments, and select the summary with the highest confidence as the initial video summary of each frame segment; Step 13: The label information carried by the object features is used to complete the initial video summary, and finally a video summary of the corresponding frame segment is obtained.

4. The agricultural video label generation method of multi-resolution analysis according to claim 3, It is characterized in that The step 11 comprises: Step 111: using the VIT model pre-trained on the Agriculture crop images dataset to extract crop features from each agricultural video frame segment to be detected. If crop features are extracted, all the extracted crop features constitute the crop feature vector to be encoded of each agricultural video frame segment to be detected, and using the constructed crop feature recognition library to label the crop feature vector to be encoded of each agricultural video frame segment to be detected, so as to obtain the crop feature vector to be encoded carrying the crop label of each agricultural video frame segment to be detected; Step 112: using the resnet50 model pre-trained on the Multi-class weather dataset to extract agricultural meteorological features from each agricultural video frame segment to be detected. If agricultural meteorological features are extracted, all the extracted agricultural meteorological features constitute the agricultural meteorological feature vector to be encoded for each agricultural video frame segment to be detected, and using the constructed agricultural meteorological feature recognition library to label the agricultural meteorological feature vector to be encoded for each agricultural video frame segment to be detected, so as to obtain the agricultural meteorological feature vector to be encoded for each agricultural video frame segment to be detected carrying the agricultural meteorological label; Step 113: using the vit model pre-trained on the Agricultural-Vision dataset to extract agricultural activity features from each agricultural video frame segment to be detected. If agricultural activity features are extracted, all the extracted agricultural activity features constitute the agricultural activity feature vector to be encoded of each agricultural video frame segment to be detected, and using the constructed agricultural activity feature recognition library to label the agricultural activity feature vector to be encoded of each agricultural video frame segment to be detected, so as to obtain the agricultural activity feature vector to be encoded carrying the agricultural activity label for each agricultural video frame segment to be detected; Step 114: Use the YOLO V8 model pre-trained on the IP102 dataset to extract agricultural pest features from each agricultural video frame segment to be detected. If agricultural pest features are extracted, all extracted agricultural pest features constitute the agricultural pest feature vector to be encoded for each agricultural video frame segment to be detected, and use the constructed agricultural pest feature recognition library to label the agricultural pest feature vector to be encoded for each agricultural video frame segment to be detected, so as to obtain the agricultural pest feature vector to be encoded for each agricultural video frame segment to be detected carrying the agricultural pest label.

5. The agricultural video label generation method of multi-resolution analysis according to claim 1, It is characterized in that The step 2 comprises: Step 21: The frame vector Z' carrying the position code obtained in step 1 multi Input into the multi-layer Transformer encoder; Step 22: Encode the crop feature vectors to be encoded, the agricultural meteorological feature vectors to be encoded, the agricultural activity feature vectors to be encoded, and the agricultural pest feature vectors to be encoded that may exist in each agricultural video frame segment to be detected into word vectors, and form a sentence vector with the same length as the word vector through sentence encoding of the initial video summary of each agricultural video frame segment to be detected; Step 23: Combine the sentence vector and word vector obtained in step 22 into an encoding vector; Step 24: The encoding vector obtained in step 23 is fed into the multi-layer Transformer decoder, and the encoding output result of the Transformer encoder obtained in step 21 is input into the current layer Transformer decoder as the supervision target to obtain the decoding output result of the current layer Transformer decoder; Step 25: Input the output result of the Transformer decoder of the current layer into the Transformer decoder of the next layer, and at the same time input the encoded output result of the Transformer encoder of the next layer as the supervision target into the Transformer decoder of this layer, and finally obtain the output result of the Transformer decoder of this layer. After N iterations, the final decoding output result is obtained; N is the number of encoder-decoder layers.

6. The agricultural video label generation method of multi-resolution analysis according to claim 1, It is characterized in that The step 3 comprises: Step 31: According to the final decoded feature vector, a plurality of pending agricultural video summaries and the confidence corresponding to each pending agricultural video summary are obtained, and finally the pending agricultural video summary with the highest confidence is determined as the multi-granularity video summary of the agricultural video to be detected; Step 32: extract keywords from the multi-granularity video summary of the agricultural video to be detected, and use the extracted keywords as storage tags of the agricultural video to be detected.

7. The agricultural video label generation method of multi-resolution analysis according to claim 6, It is characterized in that The step 31 comprises: The final decoded feature vector is respectively input into the confidence feedforward model and the LSTM model with attention mechanism to obtain multiple pending agricultural video summaries output by the LSTM model and the confidence corresponding to each pending agricultural video summary output by the confidence feedforward model. Finally, the pending video summary with the highest confidence is selected as the multi-granularity video summary of the agricultural video to be detected.

8. The agricultural video label generation method of multi-resolution analysis according to claim 7, It is characterized in that The step 32 comprises: Multi-granularity video summary keywords of the agricultural video to be detected are extracted through the RoBERTa-based word embedding and global semantic vector extraction model, and the extracted keywords are used as storage labels of the agricultural video to be detected.

9. The agricultural video label generation method of multi-resolution analysis according to claim 6 or 8, It is characterized in that The step 32 specifically includes: Step 321: Perform text preprocessing on the multi-granular video summary of the agricultural video to be detected to obtain a candidate word sequence {w 1 ,w 2 ,...,w n }, the candidate word sequence {w 1 ,w 2 ,...,w n } Input the RoBERTa word vector model to obtain the candidate semantic word vector of the video summary {v 1 ,v 2 ,...,v n }; Step 322: directly obtain the global semantic vector v of the multi-granularity video summary of the agricultural video to be detected through the global semantic vector extraction model based on RoBERTa all ; Step 323: Calculate the candidate semantic word vectors {v 1 ,v 2 ,...,v n } and the global semantic vector v all The similarity of the two keywords is calculated and the N keywords with the highest scores are selected as the final results of the extraction.

10. A device for generating agricultural video labels by multi-resolution analysis, which implements the method for generating agricultural video labels by multi-resolution analysis according to any one of claims 1 to 9, It is characterized in that include: A frame vector acquisition module is used to obtain a frame vector carrying a position code from the agricultural video to be detected; A decoding module is used to input the frame vector carrying the position code into a multi-layer encoder, to form a multi-granularity agricultural video coding vector with object features of different granularities extracted from the agricultural video to be detected and the video summary of the agricultural video to be detected, and to input the multi-granularity agricultural video coding vector into a multi-layer decoder, and the decoding output result of each layer of the decoder in the multi-layer decoder and the encoding output result of each layer of the encoder in the multi-layer encoder are used together as the input of the next layer of the decoder, and finally a final decoding feature vector after multiple decoding is obtained; The label generation module is used to obtain multiple pending agricultural video summaries and the confidence corresponding to each pending agricultural video summary based on the final decoded feature vector, determine the multi-granularity video summary of the agricultural video to be detected based on the confidence, and obtain the storage label of the agricultural video to be detected based on the multi-granularity video summary of the agricultural video to be detected.

Citation Information

Patent Citations

  • Video description generation method and device and storage medium

    CN114386260A

  • Video description generation system based on relation awareness

    CN115311598A

  • Video information abstract generation method and device, storage medium and computer equipment

    CN115757867A

  • Time sequence feature-based pseudo label generation video saliency detection method

    CN117237842A

  • Semantic feature selection and attention fusion-oriented video description generation method and system

    CN117789076A