Wide-area target searching method and device for unmanned aerial vehicle and medium

Abstract

The invention discloses a wide-area target searching method and device for an unmanned aerial vehicle and a medium, and relates to the field of unmanned aerial vehicle target searching, and the method comprises the steps: obtaining a latest video frame shot by the unmanned aerial vehicle at a current time node, segmenting the latest video frame into a plurality of image blocks, and carrying out the feature vector conversion of the image blocks; obtaining a semantic description text containing a tracking target, and matching the scene picture feature vector with scene text description; when the matching exists, acquiring a corresponding matched image block; performing target detection on the matched image blocks to obtain a detection frame, and performing feature vector conversion on the detection frame to obtain a target picture feature vector; and matching the target picture feature vector with the target text description, and determining the target with the highest matching value as a search target. A wide-area video frame is segmented into a plurality of image blocks, details and features in an image can be captured more accurately, and text and image features of a target are aligned in a feature space through matching of text feature vectors and image feature vectors.

Description

Technical Field

[0001] The present application relates to the field of unmanned aerial vehicle target search, and specifically to a method, device and medium for wide-area target search of an unmanned aerial vehicle. Background Art

[0002] With the rapid development of drone technology, its application in many fields is becoming more and more extensive. In drone pursuit scenarios, especially when performing wide-area target search missions, drones have become increasingly important search equipment due to their excellent maneuverability and precise positioning capabilities. However, wide-area target search often faces complex and changeable environments, such as undulating terrain, changing weather, dynamic movement of targets, etc., which puts higher demands on the search capabilities of drones.

[0003] Traditional wide-area target search methods mainly rely on pilots to control the flight of drones and an expert with deep professional knowledge to perform visual recognition to determine whether there is a target in the picture. This highly manual search method is not only inefficient, but also easily affected by human factors, resulting in low accuracy and reliability of search results. In addition, existing search algorithms can only search for fixed categories of targets and cannot cope with scenes with changing targets, resulting in low algorithm practicality. Summary of the invention

[0004] In order to solve the above problems, the present application proposes a method for wide-area target search of a UAV, comprising:

[0005] Obtain the latest video frame shot by the drone at the current time node, divide the latest video frame into a number of image blocks, and perform feature vector conversion on the several image blocks based on scene features to obtain corresponding scene picture feature vectors;

[0006] Acquire a semantic description text containing a tracking target, and sequentially match the scene image feature vector with the scene text description in the semantic description text;

[0007] When there is a match between the scene image feature vector and the scene text description, obtaining a corresponding matching image block, wherein the matching image block is not unique;

[0008] Performing target detection on the matched image blocks to obtain a number of detection frames containing target objects, and performing feature vector conversion on the images corresponding to the detection frames based on target features to obtain corresponding target image feature vectors;

[0009] The target image feature vector is matched with the target text description in the semantic description text, and the target object with the highest matching value is determined as the search target.

[0010] On the other hand, the present application also proposes a device for wide-area target search of a drone, comprising:

[0011] at least one processor; and,

[0012] a memory communicatively connected to the at least one processor; wherein,

[0013] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a method for wide-area target search of a drone as described in the above example.

[0014] On the other hand, the present application also proposes a non-volatile computer storage medium storing computer executable instructions, wherein the computer executable instructions are configured as: a method for wide-area target search of a drone as described in the above example.

[0015] The present application proposes a method for wide-area target search of a drone, which can bring the following beneficial effects:

[0016] By using drones to shoot wide areas and divide wide-area video frames into multiple image blocks, it can ensure that every pixel or image area is taken into account, more accurately capture the details and features in the image, and achieve better target tracking effects.

[0017] By matching text and image feature vectors, the target's text and image features are aligned in the feature space. Based on the target's description, the drone can autonomously search and lock on targets in the area. In the face of changes in the search target, there is no need to retrain the model, which enhances the scope of use of the drone's target search. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0019] Figure 1 A flowchart of a method for wide-area target search of a drone in an embodiment of the present application;

[0020] Figure 2 This is a schematic diagram of a specific process of performing target search in an embodiment of the present application;

[0021] Figure 3 This is a schematic diagram of the mounting structure of the UAV in the embodiment of the present application;

[0022] Figure 4 A schematic diagram of a device for wide-area target search of a drone in an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0024] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0025] like Figure 1 As shown, the embodiment of the present application provides a method for wide-area target search of a drone, comprising:

[0026] S101: Obtain the latest video frame shot by the drone at the current time node, divide the latest video frame into a number of image blocks, and perform feature vector conversion on the several image blocks based on scene features to obtain corresponding scene picture feature vectors.

[0027] Specifically, Figure 2 As shown, in the video data shot by the drone, the latest video frame shot by the drone at the current time node is obtained, and the data record table is queried based on the current time point to determine whether the latest video frame exists. If the latest video frame does not exist, it is determined that the latest video frame has not been matched, and the subsequent target tracking process is executed. If the latest video frame already exists, the matching status of the latest video frame in the data record table is obtained. When the matching status is unmatched, it is determined that the latest video frame has not been matched, and the subsequent target tracking process can be executed. When the matching status is matched, it is determined that the latest video frame has been matched, and the target tracking process of the video frame is no longer executed, and the latest video frame corresponding to the next time point is obtained.

[0028] Furthermore, the latest video frame is divided into several image blocks by overlapping based on a preset size, which can be set to a size of 256*256 in the present application.

[0029] By overlapping and blocking the video frames so that there is a certain overlap between adjacent image blocks, it can be ensured that each pixel or image area can be taken into account, and the details and features in the image can be captured more accurately, thereby improving the accuracy of image processing. In addition, when some image blocks are subject to noise, occlusion or damage, they can still be restored or compensated by the information in other overlapping blocks, thereby ensuring the stability and reliability of the entire image processing.

[0030] It should be noted that in the drone pursuit scenario, the drone continuously shoots video data and transmits the captured video data in real time to the image processing unit on the drone, which tracks the target in the video data. Figure 3 The figure shows the architecture diagram of the drone and its mounted high-performance recording edge computing platform.

[0031] Furthermore, several image blocks are input into the multimodal pre-training model, and based on the scene features, feature vectors of several image blocks are simultaneously converted through the image encoder in the multimodal pre-training model to obtain corresponding scene image feature vectors.

[0032] Among them, in the embodiment of the present application, the multimodal pre-training model can be a CLIP multimodal model (Contrastive Language-Image Pre-Training). When the original CLIP model is trained, the model parameters of the CLIP model are adjusted by collecting the training data set of the drone perspective, so that it is more suitable for drone tracking scenarios under wide areas. At the same time, since the original design of the CLIP model is relatively large and the resource requirements are high, it is difficult to run on the edge device of the drone, so the model needs to be lightweight. The traditional model needs to traverse each target image for encoding, and the more targets, the longer it takes. The trained CLIP model supports encoding N targets at the same time (N>0), and the input of the model is adjusted from [1,3,H,W] to [N,3,H,W], where N is the number of targets, H and W are the height and width pixels of the target image respectively, and the output is the encoding of N targets. When there are many targets, the reasoning speed of the model is greatly improved.

[0033] S102: Acquire a semantic description text containing a tracking target, and sequentially match the scene image feature vector with the scene text description in the semantic description text.

[0034] Specifically, a semantic description text containing a tracking target is obtained, the semantic description text is parsed, and the parsing result is output in a preset format, which is a "scene-text" format. The parsing result is input into a multimodal pre-training model, and a text encoder in the multimodal pre-training model is used to convert the parsing result into a feature vector to obtain a scene text feature vector and a target text feature vector.

[0035] For example, the obtained semantic description text is "the red bus in the parking lot by the lake". The semantic description text is parsed based on the preset format, and the parsed result is {"scene":{"type":"parking lot by the lake","location":{"latitude":xxx,"longitude":xxx}},"target":{"type":"vehicle","specific_description":"red bus"}}.

[0036] Furthermore, the scene image feature vectors corresponding to several image blocks are matched with the scene text feature vectors in turn, the scene cosine similarity between the scene image feature vector and the scene text feature vector is calculated, and it is determined whether the scene cosine similarity corresponding to the image block is greater than a scene preset threshold.

[0037] If it is determined that the scene cosine similarity corresponding to the image block is not greater than the scene preset threshold, it is determined that the image block does not contain the tracking target, and a non-tracking mark is added to the image block. If it is determined that the scene cosine similarity corresponding to the image block is greater than the scene preset threshold, a tracking mark is added to the image block whose scene cosine similarity is greater than the scene preset threshold to facilitate target matching of the image block with the tracking mark.

[0038] By matching the scene image feature vector in the image block with the scene text feature vector and combining it with cosine similarity judgment, we can more accurately determine which image blocks may contain the tracking target, thereby reducing mistracking, helping to more accurately locate the tracking target in complex scenes or backgrounds, and improving tracking accuracy.

[0039] S103: When there is a match between the scene image feature vector and the scene text description, a corresponding matching image block is obtained, and the matching image block is not unique.

[0040] Specifically, when there is a scene image feature vector that matches the scene text description, the corresponding matching image block is obtained based on the tracking mark, wherein the number of matching image blocks is not unique.

[0041] S104: Performing target detection on the matched image blocks to obtain a plurality of detection frames containing target objects, and performing feature vector conversion on the images corresponding to the detection frames based on target features to obtain corresponding target image feature vectors.

[0042] Specifically, the YoLov8 model is used to perform target detection on all matching image blocks to obtain several detection frames containing target objects, and the detection frames are cropped to obtain a target image containing the target object.

[0043] Furthermore, the target image is input into the multimodal pre-trained model, and based on the target features, the multimodal pre-trained model is used to convert the feature vectors of the target objects in the image, and the target image feature vectors corresponding to the target objects in the image blocks are matched respectively.

[0044] It should be noted that the architecture of the YOLOv8 model mainly includes the backbone network (Backbone), the neck network (Neck), and the detection head (Head). The backbone is used to extract features in matching image blocks, the neck network is used to aggregate the extracted features of different scales, and the detection head is used for target detection, including the prediction of the detection box position, the prediction of the target category, and the calculation of the confidence of the detection box position and the target.

[0045] S105: Match the target image feature vector with the target text description in the semantic description text, and determine the target object with the highest matching value as the search target.

[0046] Specifically, the number of target objects in the matching image blocks is determined. When the number of target objects is not greater than 1, the target cosine similarity between the target image feature vector and the target text feature vector corresponding to the target object is calculated, and it is determined whether the target cosine similarity is greater than the target preset threshold. If the target cosine similarity is not greater than the target preset threshold, it is determined that the latest video frame does not contain the tracking target, and the latest video frame at the next time point is obtained. If the target cosine similarity is greater than the target preset threshold, the target object is determined to be a tracking target.

[0047] When the number of target objects is greater than 1, the target cosine similarities of the target image feature vector and the target text feature vector corresponding to the target object are calculated respectively, and the highest value target cosine similarity is obtained. It is determined whether the highest value cosine similarity is greater than the target preset threshold. If not, it is determined that the latest video frame does not contain the tracking target, and the latest video frame at the next time point is obtained. If so, the target object corresponding to the highest value cosine similarity is determined to be the tracking target.

[0048] Furthermore, after the tracking target is determined, the tracking target is tracked by a target tracking algorithm. In the embodiment of the present application, a multi-target tracking algorithm (DeepSORT algorithm) is used for tracking.

[0049] It should be noted that when the drone has entered the scene where the target is located, the focus of the drone camera is adjusted to enlarge the search area to facilitate obtaining a clear image of the target. When the focus is adjusted in place, the target tracking model is started, all detected targets in the picture are obtained and the target image is cropped, and the image encoder is called again to encode all tracked target images, and the cosine similarity is calculated with the target text feature vector. If it meets the preset threshold, the target is successfully searched; otherwise, a new video frame is obtained to continue target tracking.

[0050] Furthermore, a matching record of the latest video frame is obtained, the matching record includes a shooting time, a matching result and a matching status, and the matching record of the latest video frame is added to the data record table.

[0051] It should be noted that in the embodiment of the present application, the YoLov8 model is trained under the drone perspective data for four common targets: vehicles, pedestrians, electric bicycles and motorcycles, and combined with DeepSORT for target tracking. After the model training is completed, the traditional method does not have the ability to further determine the target. For example, it can only search for "vehicles" but not "red vehicles"; because the drone is equipped with only one camera, there are significant limitations in directly searching for "red buses in the lakeside parking lot". Although directly zooming in on the target area can highlight the details of the target, it will result in a narrowing of the field of view and loss of surrounding information of the scene where the target is located, making it difficult to confirm whether it is located in the designated "lakeside parking lot". In addition, when the target is cropped and then the CLIP model is used for image-text matching, the cropped image usually only contains the target itself, and the scene features of the square cannot be retained, affecting the accuracy of positioning.

[0052] By using drones to shoot wide areas and divide wide-area video frames into multiple image blocks, it can ensure that every pixel or image area is taken into account, more accurately capture the details and features in the image, and achieve better target tracking effects.

[0053] By matching text and image feature vectors, the target's text and image features are aligned in the feature space. Based on the target's description, the drone can autonomously search and lock on targets in the area. In the face of changes in the search target, there is no need to retrain the model, which enhances the scope of use of the drone's target search.

[0054] like Figure 4 As shown, the embodiment of the present application also proposes a device for wide-area target search of a drone, including:

[0055] at least one processor; and,

[0056] a memory communicatively connected to the at least one processor; wherein,

[0057] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a method for wide-area target search of a drone as described in any of the above embodiments.

[0058] An embodiment of the present application further provides a non-volatile computer storage medium storing computer executable instructions, wherein the computer executable instructions are configured to be: a method for wide-area target search of a drone as described in any of the above embodiments.

[0059] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0060] The devices and media provided in the embodiments of the present application correspond one-to-one to the methods. Therefore, the devices and media also have similar beneficial technical effects as the corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0061] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0062] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0063] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0064] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0065] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0066] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0067] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0068] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0069] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A method for wide-area target search of an unmanned aerial vehicle, characterized in that: include: Obtain the latest video frame shot by the drone at the current time node, divide the latest video frame into a number of image blocks, and perform feature vector conversion on the several image blocks based on scene features to obtain corresponding scene picture feature vectors; Acquire a semantic description text containing a tracking target, and sequentially match the scene image feature vector with the scene text description in the semantic description text; When there is a match between the scene image feature vector and the scene text description, obtaining a corresponding matching image block, wherein the matching image block is not unique; Performing target detection on the matched image blocks to obtain a number of detection frames containing target objects, and performing feature vector conversion on the images corresponding to the detection frames based on target features to obtain corresponding target image feature vectors; The target image feature vector is matched with the target text description in the semantic description text, and the target object with the highest matching value is determined as the search target.

2. The method for wide-area target search of an unmanned aerial vehicle according to claim 1, characterized in that: The step of dividing the latest video frame into a plurality of image blocks and converting the plurality of image blocks into feature vectors based on scene features to obtain corresponding scene picture feature vectors specifically includes: Based on a preset size and a preset overlap amount, overlappingly segmenting the latest video frame to obtain a plurality of picture blocks; The plurality of image blocks are input into a multimodal pre-trained model, and based on scene features, feature vectors are simultaneously converted to the plurality of image blocks through an image encoder in the multimodal pre-trained model to obtain corresponding scene image feature vectors respectively.

3. The method for wide-area target search of an unmanned aerial vehicle according to claim 2, characterized in that: Before sequentially matching the scene image feature vector with the scene text description in the semantic description text, the method further includes: Parsing the semantic description text and outputting the parsing result in a preset format; The parsing result is input into the multimodal pre-training model, and the parsing result is converted into a feature vector by a text encoder in the multimodal pre-training model to obtain a scene text feature vector and a target text feature vector.

4. The method for wide-area target search of an unmanned aerial vehicle according to claim 3, characterized in that: The sequentially matching the scene image feature vector with the scene text description in the semantic description text specifically includes: Sequentially matching the scene picture feature vectors corresponding to the plurality of image blocks with the scene text feature vector, and calculating the scene cosine similarity between the scene picture feature vector and the scene text feature vector; Determine whether the scene cosine similarity is greater than a scene preset threshold; If not, it is determined that the image block to be matched does not contain the tracking target, and a non-tracking mark is added to the image block to be matched; If yes, a tracking mark is added to the image block whose scene cosine similarity is greater than a preset scene threshold.

5. The method for wide-area target search of an unmanned aerial vehicle according to claim 3, characterized in that: The matching of the target image feature vector with the target text description in the semantic description text, and determining the target object with the highest matching value as the search target, specifically includes: Determining the number of target objects in the matched image blocks; When the number of the target objects is not greater than 1, the target cosine similarity between the target image feature vector and the target text feature vector corresponding to the target object is calculated, and it is determined whether the target cosine similarity is greater than a target preset threshold; If not, it is determined that the latest video frame does not contain the tracking target, and the latest video frame at the next time point is continuously acquired; If so, the target object is determined to be a tracking target.

6. The method for wide-area target search of an unmanned aerial vehicle according to claim 5, characterized in that: After determining the number of target objects in the matching image blocks, the method further includes: When the number of target objects is greater than 1; Calculate the target cosine similarity of the target image feature vector and the target text feature vector corresponding to the target object, obtain the highest value of the target cosine similarity, and determine whether the highest value of the cosine similarity is greater than a target preset threshold; If not, it is determined that the latest video frame does not contain the tracking target, and the latest video frame at the next time point is continuously acquired; If so, the target object corresponding to the highest cosine similarity is determined to be the tracking target.

7. The method for wide-area target search of an unmanned aerial vehicle according to claim 1, characterized in that: After matching the target image feature vector with the target text description in the semantic description text and determining that the target object with the highest matching value is the search target, the method further includes: Obtaining a matching record of the latest video frame, wherein the matching record includes a shooting time, a matching result, and a matching status; Add the matching record of the latest video frame to the data record table.

8. The method for wide-area target search of an unmanned aerial vehicle according to claim 7, characterized in that: Before dividing the latest video frame into a plurality of image blocks, the method further comprises: Based on the current time point, query the data record table to determine whether the latest video frame exists; If not, it is determined that the latest video frame does not match; If yes, obtaining the matching status of the latest video frame; When the matching state is unmatched, determining that the latest video frame is unmatched; When the matching state is matched, it is determined that the latest video frame has been matched, and the latest video frame corresponding to the next time point is obtained.

9. A device for wide-area target search of an unmanned aerial vehicle, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform, for example: Obtain the latest video frame shot by the drone at the current time node, divide the latest video frame into a number of image blocks, and perform feature vector conversion on the several image blocks based on scene features to obtain corresponding scene picture feature vectors; Acquire a semantic description text containing a tracking target, and sequentially match the scene image feature vector with the scene text description in the semantic description text; When there is a match between the scene image feature vector and the scene text description, obtaining a corresponding matching image block, wherein the matching image block is not unique; Performing target detection on the matched image blocks to obtain a number of detection frames containing target objects, and performing feature vector conversion on the images corresponding to the detection frames based on target features to obtain corresponding target image feature vectors; The target image feature vector is matched with the target text description in the semantic description text, and the target object with the highest matching value is determined as the search target.

10. A non-volatile computer storage medium storing computer executable instructions, characterized in that: The computer executable instructions are configured to: Obtain the latest video frame shot by the drone at the current time node, divide the latest video frame into a number of image blocks, and perform feature vector conversion on the several image blocks based on scene features to obtain corresponding scene picture feature vectors; Acquire a semantic description text containing a tracking target, and sequentially match the scene image feature vector with the scene text description in the semantic description text; When there is a match between the scene image feature vector and the scene text description, obtaining a corresponding matching image block, wherein the matching image block is not unique; Performing target detection on the matched image blocks to obtain a number of detection frames containing target objects, and performing feature vector conversion on the images corresponding to the detection frames based on target features to obtain corresponding target image feature vectors; The target image feature vector is matched with the target text description in the semantic description text, and the target object with the highest matching value is determined as the search target.

Images