A drone video retrieval method based on deep learning cross-modal technology

Abstract

The present invention discloses a drone video retrieval method based on deep learning cross-modal technology, comprising: performing preliminary classification based on user-entered text to obtain the time, place, object, and purpose to be detected; pre-screening the original video from the three dimensions of time, place, and object to obtain possible key frames and target frames containing the detected object; accurately matching the text with the pre-screened key frames based on an improved VIT model to obtain accurate image frames; building a GRU-CRF-based neural network, adjusting the step size of the GRU network according to the purpose to be detected, processing the accurate image frames, and classifying image frames belonging to the same video clip to obtain video clips that meet the user's requirements. The present invention extracts key frames and classifies image frames based on information from four dimensions, greatly shortening video retrieval time and improving video retrieval efficiency and accuracy.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and drone technology, and in particular to a drone video retrieval method based on deep learning cross-modal technology. Background Art

[0002] Traditional text-based video retrieval methods are mainly divided into direct retrieval and indirect retrieval. The core of direct retrieval methods is to accurately match the text query entered by the user with the external descriptive information of the video (such as title, subtitles, tags, etc.).

[0003] Indirect retrieval methods aim to bridge the gap between textual and visual content. They first evenly segment the video content into multiple time frames. Advanced image recognition techniques are then used to convert each frame into quantifiable image features. Natural language processing (NLP) techniques are then used to convert the text query into a corresponding feature vector, enabling cross-modal matching between textual and image features.

[0004] Direct retrieval methods lack a deep understanding of video content. Their matching process is essentially limited to text-to-text, failing to capture visual features, emotional expressions, and dynamic scene changes within the video. This limits the comprehensiveness and accuracy of retrieval results. Indirect retrieval's uniform frame-splitting strategy results in processing efficiency that is directly related to video length, limiting its application when processing large-scale or long video data.

[0005] With the increasing popularity of drone technology and the expansion of its applications, the amount of drone video data is growing exponentially. These videos contain a wealth of information, and efficient management and retrieval of this data has become a pressing challenge. Considering a 30-minute video with a frame rate of 30 frames per second, a 30-minute video can be divided into 54,000 images. Similarly, drones store thousands of videos during routine flight, and the number of images involved reaches hundreds of millions. Consequently, traditional video frame and text matching methods require prohibitive computing power and time.

[0006] In recent years, deep learning has made significant progress in the fields of computer vision and natural language processing. Deep learning models can autonomously learn feature representations of text and images, and understand and analyze complex data, providing new ideas for cross-modal video retrieval. Summary of the Invention

[0007] To address the problems that direct retrieval methods cannot learn video content features and indirect retrieval takes a long time, the present invention provides a drone video retrieval method based on deep learning cross-modal technology.

[0008] In order to achieve the above technical objectives, the technical solution adopted by the present invention is:

[0009] A method for retrieval of drone videos based on deep learning cross-modal technology, the method comprising the following steps:

[0010] S1, performs feature vectorization processing on the search text and converts the search text into a high-dimensional vector; performs preliminary classification based on the text input by the user to obtain the time, place, object and purpose to be detected;

[0011] S2, based on the detection object information extracted from the continuous frame transformation of the video, the drone flight parameters and the search text entered by the user, pre-screens the original video from three dimensions: time, location, and object, and obtains possible key frames and target boxes containing the detection objects;

[0012] S3, improving the VIT model using the target frame obtained in step S2, and accurately matching the text and the pre-screened key frames based on the improved VIT model to obtain an accurate image frame;

[0013] S4: Build a GRU-CRF-based neural network, adjust the step size of the GRU network according to the purpose to be detected obtained in step S1, use the GRU-CRF-based neural network to process the accurate picture frames obtained in step S3, classify the picture frames belonging to the same video clip, and obtain a video clip that meets the user's requirements.

[0014] Step S1 further comprises:

[0015] Perform data cleaning on the input search text to remove noise and irrelevant information in the search text;

[0016] The cleaned search text is segmented using the word segmentation library, and each word is mapped to a fixed-size real number vector through the embedding layer;

[0017] A pre-classification model is built based on a neural network, which includes a multi-scale convolutional neural network, an LSTM network, an attention module and four fully connected layers connected in sequence; the multi-scale convolutional neural network extracts multi-scale features from the input real number vector; the LSTM network processes the multi-scale features extracted by the multi-scale convolutional neural network to capture contextual semantics; the attention module focuses on key information in the contextual semantics and assigns a higher weight to the focused information; the four fully connected layers act as classifiers, process the sentence features output by the attention module, and output the time, place, object and purpose to be detected respectively.

[0018] Step S2 further comprises:

[0019] S21, filtering out drone video clips that meet the time dimension requirement based on the drone flight shooting time, filtering out drone video clips that meet the time requirement based on the pixel transformation of consecutive video frames, and eliminating pictures of the same scene;

[0020] S22, confirming the detection range of the image frames selected in step S21 based on the drone's GPS, flight altitude, and lens focal length, and selecting image frames that meet the location dimension requirements;

[0021] S23, using the Yolo target detection model to detect the picture frames filtered out in step S22, retaining the picture frame containing the detection object and the target box P [upper left corner coordinate, lower left corner coordinate, upper right corner coordinate, lower right corner coordinate] containing the detection object in the picture.

[0022] Step S2 further comprises:

[0023] Before pre-screening, the image is converted into a grayscale image through Opencv and compared with the grayscale image of the previous frame. The absolute difference is calculated and binarized to obtain a binary image representing the difference between the previous and next frames. The number of non-zero pixels in the binary image is counted. If the count exceeds the preset minimum number of difference pixels, it is considered that there is a difference between the current frame and the previous frame, and the current image is saved.

[0024] Step S22 further includes:

[0025] S221, calculate the camera spatial resolution:

[0026] ;

[0027] Where ratio represents the spatial resolution, S represents the pixel pitch, H represents the flight altitude of the drone, and f represents the focal length of the lens;

[0028] In step S222, the camera coverage is determined based on the drone's GPS, flight altitude, and camera spatial resolution. The effective pixel area is set at the center of the photo (a, b). The effective detection area is set at the center of the drone's GPS (a*ratio, b*ratio). Image frames whose detection area does not include the detection location are discarded.

[0029] Step S3 further comprises:

[0030] The original image is divided into multiple image blocks. The image block where the target box P is located is assigned an initial weight w_high. The initial weight of the image block outside the target box is set to w_low, w_high>w_low; the final weight w of the image block is set according to the overlap ratio between the image block and the target box. i , the weight setting formula is:

[0031] ;

[0032] Where r i Indicates the ratio of the overlapping area between the image block i and the target box P to the area of the image block;

[0033] In the initialization phase, the set weights are multiplied by the embedding vector of the image block, and the embedding vector of the image block where the target frame P is located is weighted. By learning the image features of the target area, the vector representation of the image frame is output; the similarity between the vector representation V of the image frame and the vector representation T of the retrieved text obtained in step S1 is calculated:

[0034] ;

[0035] Where, represents the dot product of vectors T and V, and are the Euclidean lengths of vectors T and V respectively, The cosine similarity between the vector representation V of the image frame and the vector representation T of the retrieved text;

[0036] A similarity threshold τ is set. If the calculated similarity is greater than or equal to the similarity threshold τ, the image frame is considered to be similar in content to the retrieved text and is saved as an accurate image frame.

[0037] Step S4 further comprises:

[0038] Analyze the detection target and identify its corresponding time requirement granularity and event characteristics. Based on the time requirement granularity, set the GRU network step size and input accurate image frames into the GRU network. Using the GRU network's temporal modeling capabilities, output image frame sequences with corresponding event feature labels for different time periods, completing preliminary temporal feature extraction and event labeling.

[0039] The image frame sequence with corresponding event feature labels is imported into the CRF model. Based on the contextual association between the image frames and the dependency between the labels, the CRF model combines the scene changes and action continuity factors at different time points to calculate the joint probability of the three different time labels (event start, event in progress, and event end) over the entire image frame sequence.

[0040] Based on the joint probability, it determines whether the event duration meets the preset time requirements, and outputs the feature representation and label information corresponding to each image frame to optimize the event timing analysis results;

[0041] According to the original time sequence of the video, the picture frames belonging to the same video clip are classified and integrated to obtain video clips containing events, themes or attributes corresponding to the detection purpose, and the video content is structurally extracted and divided.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] First, the drone video retrieval method of the present invention, based on deep learning cross-modal technology, synchronously obtains information in four dimensions of time, place, object and purpose to be detected from the retrieval text input by the user, extracts key frames and classifies the key frames based on the information in these four dimensions, greatly shortening the video retrieval time and improving the video retrieval efficiency and accuracy.

[0044] Second, the drone video retrieval method based on deep learning cross-modal technology of the present invention can adaptively process different detection scenarios, meet different application requirements, and has good universality.

[0045] Third, the drone video retrieval method based on deep learning cross-modal technology of the present invention can quickly complete video retrieval, so that the drone fleet can adjust the flight strategy in time according to the video retrieval results. It is especially suitable for complex and changeable scenarios, such as traffic control on roads with high traffic volume, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is a flow chart of the drone video retrieval method based on deep learning cross-modal technology of the present invention;

[0047] Figure 2 Schematic diagram of the pre-classification model structure;

[0048] Figure 3 This is a schematic diagram of the camera's effective area calculation principle;

[0049] Figure 4 This is the principle diagram of GRU-CRF neural network detection. DETAILED DESCRIPTION

[0050] The embodiments of the present invention are described in further detail below with reference to the accompanying drawings.

[0051] The present invention discloses a method for retrieval of unmanned aerial vehicle (UAV) videos based on deep learning cross-modal technology, the method comprising the following steps:

[0052] S1, performs feature vectorization processing on the search text and converts the search text into a high-dimensional vector; performs preliminary classification based on the text input by the user to obtain the time, place, object and purpose to be detected;

[0053] S2. Pre-screen the original video from the three dimensions of time, location, and object according to the detection object information extracted from the continuous frame transformation of the video, the flight parameters of the drone, and the retrieved text input by the user, and obtain possible key frames and target boxes containing the detection objects;

[0054] S3. Improve the VIT model through the target boxes obtained in step S2, and accurately match the text and picture frames according to the improved VIT model to obtain accurate picture frames;

[0055] S4. Build a neural network based on GRU-CRF, adjust the step size of GRU according to the purpose to be detected obtained in step S1, and process the accurate picture frames obtained in step S3 using the neural network based on GRU-CRF to classify the picture frames belonging to the same video segment and obtain the video segment that meets the user's requirements.

[0056] Step S1 specifically includes:

[0057] Text data cleaning. Remove the noise and irrelevant information in the text, such as HTML tags, JavaScript code, special characters (such as line breaks, tab characters, etc.), numbers, extra spaces, etc. Remove common stop words (such as "of", "is", "in", etc.), which contribute little to text analysis but consume computing resources. Convert all text to lowercase to eliminate the processing difficulty caused by case differences.

[0058] Word segmentation and word embedding. Segment the retrieved text through a word segmentation library, and then map each token to a real number vector of a fixed size through an embedding layer. These vectors will be continuously adjusted during the training process to better capture the semantic relationships between tokens.

[0059] Build a neural network based on multi-scale CNN-LSTM-Attention to pre-classify the text input by the user and obtain the "detection time, detection location, detection object, and detection purpose" in the text. The neural network architecture is as Figure 2 shown.

[0060] Description of the neural network process and dimensional transformation:

[0061] (1) Input layer

[0062] The input is a text vector, assuming its dimension is (batch_size, sequence_length, embedding_size). Among them, batch_size represents the number of text samples input at one time; sequence_length represents the length of the text sequence, that is, the number of characters in the text; embedding_size is the vector dimension corresponding to each character, that is, the word embedding dimension.

[0063] (2) Multi-scale CNN layer

[0064] The text vectors are fed into convolutional neural network branches with different kernel sizes (1x1, 3x3, and 5x5). The convolution operation extracts local features of varying scales by sliding the kernel across the text sequence. Smaller kernels (1x1) capture finer-grained features, while larger kernels (5x5) capture broader contextual features.

[0065] Assume the input dimensions are (batch_size, sequence_length, embedding_size). After the convolution operation, the output dimensions of each branch become (batch_size, num_filters, new_sequence_length). Here, num_filters is the number of output channels corresponding to each convolution kernel, and new_sequence_length is the length of the text sequence after the convolution operation. The results of these three branches are then concatenated along the channel dimension, resulting in a dimension of (batch_size, 3 x num_filters, new_sequence_length).

[0066] (3) LSTM layer

[0067] Taking the features output by the multi-scale CNN layer as input, the LSTM unit sequentially processes the information of each time step in the sequence. Through its memory unit and gating mechanism, it captures the long-term dependencies and contextual semantics in the text sequence.

[0068] The input dimension is (batch_size, 3 x num_filters, new_sequence_length). After LSTM processing, the output dimension becomes (batch_size, new_sequence_length, num_hidden_units), where num_hidden_units is the number of hidden units in the LSTM. The result of the last time step is taken as the output, and the dimension becomes (batch_size, num_hidden_units).

[0069] (4) Attention layer

[0070] The output of the LSTM layer is used as input, and attention scores are calculated to focus on the key information in the input features. Different weights are assigned to different features to highlight the information that is important to the classification task and suppress irrelevant information.

[0071] The input dimension is (batch_size, num_hidden_units). After the attention calculation, the output dimension remains the same, still (batch_size, num_hidden_units), but the features are now weighted according to their importance.

[0072] (5) Fully connected layer

[0073] The features output by the attention module are fed into four independent fully connected layers. The neurons in a fully connected layer are connected to all neurons in the previous layer. The input features are linearly transformed using a weight matrix, and nonlinearity is introduced using an activation function to map the features to the corresponding classification space.

[0074] The input dimensions are (batch_size, num_hidden_units). For the four tasks of detecting time, location, object, and purpose, assuming the number of classifications is t, l, o, and p respectively, the final output dimensions are (batch_size, t), (batch_size, l), (batch_size, o), and (batch_size, p), corresponding to the classification results of the four detection tasks respectively.

[0075] For example, if the user inputs the text "Please find video clips of traffic congestion lasting more than 10 minutes on Dingshan Road on November 1, 2024", the neural network will get "Detection time: November 1, 2024", "Detection location: Dingshan Road", "Detection object: vehicle congestion", and "Detection purpose: more than 10 minutes".

[0076] Step S2 specifically includes:

[0077] Based on the pixel transformation of consecutive video frames, drone video clips that meet the time requirements are screened and images of the same scene are eliminated. First, the image is converted to a grayscale image using OpenCV and compared with the grayscale image of the previous frame. Then, the absolute difference is calculated and binarized to obtain a binary image representing the difference between the previous and next frames. Finally, the number of non-zero pixels in the binary image is counted, i.e., the number of difference pixels. If the number of difference pixels exceeds a preset minimum, the current frame is considered different from the previous frame and the current image is saved.

[0078] Based on the detection object obtained in step S1, the search scope is further narrowed down by combining the drone flight parameters. The drone flight parameters include "shooting time, drone GPS, flight altitude, and lens focal length." The "shooting time" parameter is used to filter out image frames that meet the time dimension requirements, and the "drone GPS, flight altitude, lens focal length" and "detection location" parameters are used to filter out image frames that meet the spatial dimension requirements. Figure 3, taking the vertical shooting of a drone as an example, the spatial dimension screening process is as follows:

[0079] Calculate the camera spatial resolution. In vertical shooting scenarios, the ground distance and the image plane distance have a similar proportional relationship, as shown in formula (1), where ratio represents the spatial resolution, S represents the pixel pitch, H represents the drone's flight altitude, and f represents the lens focal length.

[0080] (1);

[0081] The camera coverage is determined based on the drone's GPS, flight altitude, and camera spatial resolution. The effective pixel area is set to [a, b] in the center of the photo. The effective detection area is then defined as the area with a spatial range of [a*ratio, b*ratio] centered on the drone's GPS. Image frames that do not contain the "detection location" are discarded.

[0082] A traditional small target detection model is used to quickly detect the image frames obtained above, and the image frames containing the "detection object" are retained. For example, the detection object of the traffic congestion task is "vehicle". This application uses the Yolo target detection model to detect whether there is a "car" target in the image frame, and retains the image frames with "car" and the target box P [upper left corner coordinates, lower left corner coordinates, upper right corner coordinates, lower right corner coordinates] of the "car" in the image.

[0083] Step S3 specifically includes:

[0084] According to the target frame P of the detection object obtained in step S2, a higher weight is given to the picture block Pitch containing the detection object, the image features of the target area are learned, and the vector representation of the picture frame is output. Considering that the target occupies a smaller picture area than the ground perspective under the high-altitude shooting perspective of the drone, that is, there are many invalid areas in the picture. The traditional VIT model divides the original picture into picture blocks Pitch, and then performs operations such as feature extraction. The picture blocks that do not contain the detection object will interfere with the feature learning of the model, thereby affecting the running time of the model. Therefore, the present invention combines the obtained target area P, and when the area P is in the divided Pitch, it gives the Pitch a higher initial weight, thereby guiding the model to learn the features of the target area.

[0085] The weight allocation strategy adopted by this invention is position-based weight allocation. Specifically, the image blocks within the target box P are first assigned a higher initial weight value, such as w_high, while the initial weight of the image blocks outside the target box is w_low (w_high>w_low). The weight is set according to the overlap ratio between the image block and the target box. The higher the overlap ratio, the greater the weight. Assume that the overlap area of the image block i and the target box P accounts for the ratio of the image block area to the target box area. , then the weight of image block i can be set as:

[0086] (2).

[0087] In the improved VIT model, during the initialization phase, the weights set above are multiplied by the embedding vector of the image block to weight the embedding vector of the image block where the target frame P is located. The similarity between the search text and the image frame is determined using a similarity comparison function. The vector representation T of the search text is obtained according to step S1, and the vector representation V of the image frame is obtained based on the improved VIT model. The present invention uses cosine similarity to compare the similarity between the two vectors, as shown in Formula (3):

[0088] (3);

[0089] Where, represents the dot product of vectors T and V, and are the Euclidean lengths (modulus) of vectors T and V respectively.

[0090] Set a similarity threshold τ (usually a number between 0 and 1). If the calculated similarity is greater than or equal to this threshold, as shown in formula (4), the image frame is considered to be similar in content to the search text and is saved as a candidate search result.

[0091] Similarity ≧ τ (4).

[0092] Step S4 specifically includes:

[0093] Build a GRU-CRF based neural network to classify image frames. The network structure is as follows Figure 4 As shown in Figure 2, the GRU network is used to effectively process sequence data, capture long-term dependencies, and extract dynamic semantic features. The CRF model is used to model dependencies between labels and achieve global optimal decoding to improve classification accuracy and enhance model robustness.

[0094] First, a thorough analysis of the detection target is performed to identify the corresponding fine-grained temporal requirements and event characteristics. Based on the fine-grained temporal requirements, the step size of the GRU (Gated Recurrent Unit) network is appropriately set, and the precisely selected image frames are fed into the GRU network. Leveraging the GRU network's temporal modeling capabilities, a sequence of image frames representing different time periods and labeled with corresponding event characteristics is output, completing preliminary temporal feature extraction and event labeling. The present invention dynamically sets the step size of the GRU network based on the detection objective obtained from the user input text in step S1. By dynamically adjusting the step size of the GRU network based on different detection objectives, the present invention enables the model to adapt to different detection scenarios and requirements. The core function of the GRU network is to process sequential data and capture temporal dependencies within the data. Based on the GRU network, the step size of the GRU network is adjusted based on the detection objective obtained in step S1. Specifically, the detection objective represents the fine-grained temporal requirement. For example, "10 minutes" is finer-grained than "1 hour," and the step size set for the GRU network is correspondingly smaller.

[0095] The labeled image frame sequence is then fed into a CRF (Conditional Random Field) model. The CRF model fully considers the contextual relationships between frames and the dependencies between labels. It incorporates factors such as scene changes and action continuity at different time points to calculate the joint probability of different time labels, such as "event start," "event in progress," and "event end," across the entire frame sequence. Based on these probabilities, it determines whether the event duration meets the preset time requirements and outputs feature representations and precise label information for each frame, further optimizing the event timing analysis results. The CRF model is a probabilistic graphical model. When processing a frame sequence, it considers the joint probability distribution of the entire sequence, rather than calculating it based on individual time steps like the GRU network. It combines the features output by the GRU network, taking into account the contextual relationships between frames and the dependencies between labels. For example, for the detection objective of "detecting congestion exceeding one hour," the CRF model combines the features of the frames output by the GRU network at different times and analyzes whether the changes in these features over time are consistent with a pattern of sustained activity exceeding one hour. It does not look at each picture frame in isolation, but starts from the entire sequence. Based on factors such as scene changes and action continuity at different time points, it calculates the joint probability of different time labels (such as the start of the activity, the ongoing activity, the end of the activity, etc.) on the entire picture frame sequence, so as to accurately determine whether the activity exceeds 1 hour.

[0096] Finally, image frames belonging to the same video clip are grouped and integrated according to the original video time sequence. This yields video clips that capture specific events, themes, or possess certain attributes, enabling structured extraction and effective segmentation of video content. After processing using the GRU-CRF neural network, feature representations and label information corresponding to each image frame are obtained. Based on this information, image frames belonging to the same video clip are grouped together. The resulting video clips meet user requirements. These clips can be collections of video content that capture specific events, themes, or possess certain attributes, meeting user needs for video retrieval and analysis.

[0097] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application.

[0098] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. A drone video retrieval method based on deep learning cross-modal technology, characterized by: The method comprises the following steps: S1, performs feature vectorization processing on the search text and converts the search text into a high-dimensional vector; performs preliminary classification based on the text input by the user to obtain the time, place, object and purpose to be detected; S2, based on the detection object information extracted from the continuous frame transformation of the video, the drone flight parameters and the search text entered by the user, pre-screens the original video from three dimensions: time, location, and object, and obtains possible key frames and target boxes containing the detection objects; S3, improving the VIT model using the target frame obtained in step S2, and accurately matching the text and the pre-screened key frames based on the improved VIT model to obtain an accurate image frame; S4, building a GRU-CRF-based neural network, adjusting the step size of the GRU network according to the target to be detected obtained in step S1, using the GRU-CRF-based neural network to process the accurate picture frames obtained in step S3, classifying the picture frames belonging to the same video clip, and obtaining a video clip that meets the user's requirements; step S4 further includes: Analyze the detection target and clarify its corresponding time requirement granularity and event characteristics. Based on the time requirement granularity, set the GRU network step size, input accurate image frames into the GRU network, and use the GRU network's time series modeling capabilities to output image frame sequences with corresponding event feature labels at different time periods, completing preliminary time series feature extraction and event labeling. The image frame sequence with the corresponding event feature label is imported into the CRF model. Based on the contextual association between the image frames and the dependency between the labels, the CRF model combines the scene changes and action continuity factors at different time points to calculate the joint probability of the three different time labels of event start, event in progress, and event end over the entire image frame sequence. Based on the joint probability, it determines whether the event duration meets the preset time requirements, and outputs the feature representation and label information corresponding to each image frame to optimize the event timing analysis results; According to the original time sequence of the video, the picture frames belonging to the same video clip are classified and integrated to obtain video clips containing events, themes or attributes corresponding to the detection purpose, and the video content is structurally extracted and divided.

2. The UAV video retrieval method based on deep learning cross-modal technology according to claim 1 is characterized in that: Step S1 further comprises: Perform data cleaning on the input search text to remove noise and irrelevant information in the search text; The cleaned search text is segmented using the word segmentation library, and each word is mapped to a fixed-size real number vector through the embedding layer; A pre-classification model is built based on a neural network, which includes a multi-scale convolutional neural network, an LSTM network, an attention module and four fully connected layers connected in sequence; the multi-scale convolutional neural network extracts multi-scale features from the input real number vector; the LSTM network processes the multi-scale features extracted by the multi-scale convolutional neural network to capture contextual semantics; the attention module focuses on key information in the contextual semantics and assigns a higher weight to the focused information; the four fully connected layers act as classifiers, process the sentence features output by the attention module, and output the time, place, object and purpose to be detected respectively.

3. The UAV video retrieval method based on deep learning cross-modal technology according to claim 1 is characterized in that: Step S2 further comprises: S21, filtering out drone video clips that meet the time dimension requirement based on the drone flight shooting time, filtering out drone video clips that meet the time requirement based on the pixel transformation of consecutive video frames, and eliminating pictures of the same scene; S22, confirming the detection range of the image frames selected in step S21 based on the drone's GPS, flight altitude, and lens focal length, and selecting image frames that meet the location dimension requirements; S23, using the Yolo target detection model to detect the picture frames filtered out in step S22, retaining the picture frame containing the detection object and the target box P [upper left corner coordinate, lower left corner coordinate, upper right corner coordinate, lower right corner coordinate] containing the detection object in the picture.

4. The drone video retrieval method based on deep learning cross-modal technology according to claim 3 is characterized in that: Step S2 further comprises: Before pre-screening, the image is converted into a grayscale image through Opencv and compared with the grayscale image of the previous frame. The absolute difference is calculated and binarized to obtain a binary image representing the difference between the previous and next frames. The number of non-zero pixels in the binary image is counted. If the count exceeds the preset minimum number of difference pixels, it is considered that there is a difference between the current frame and the previous frame, and the current image is saved.

5. The UAV video retrieval method based on deep learning cross-modal technology according to claim 3 is characterized in that: Step S22 further includes: S221, calculate the camera spatial resolution: Where ratio represents the spatial resolution, S represents the pixel pitch, H represents the flight altitude of the drone, and f represents the focal length of the lens; S222: The camera coverage is obtained based on the drone GPS, flight altitude, and camera spatial resolution. The area [a, b] in the center of the photo is set as the valid pixel area, and the area with a spatial range of [a*ratio, b*ratio] centered on the drone GPS is set as the valid detection area. Image frames whose detection area does not include the detection location are discarded.

6. The drone video retrieval method based on deep learning cross-modal technology according to claim 1 is characterized in that: Step S3 further comprises: The original image is divided into multiple image blocks. The image block where the target box P is located is given an initial weight w_high. The initial weight of the image block outside the target box is set to w_low, w_high>w_low; the final weight w of the image block is set according to the overlap ratio between the image block and the target box. i , the weight setting formula is: w i =r i ×(w_high-w_low)+w_low; Where r i Indicates the ratio of the overlapping area between the image block i and the target box P to the area of the image block; In the initialization phase, the set weights are multiplied by the embedding vector of the image block, and the embedding vector of the image block where the target frame P is located is weighted. By learning the image features of the target area, the vector representation of the image frame is output; the similarity between the vector representation V of the image frame and the vector representation T of the retrieved text obtained in step S1 is calculated: Where T·V represents the dot product of vectors T and V, ||T|| and ||V|| are the Euclidean lengths of vectors T and V, respectively, and Similarity represents the cosine similarity between the vector representation V of the image frame and the vector representation T of the retrieved text. A similarity threshold τ is set. If the calculated similarity is greater than or equal to the similarity threshold τ, the image frame is considered to be similar in content to the retrieved text and is saved as an accurate image frame.

Images