UAV inspection method, device, UAV, and storage medium based on SwinTrack

Abstract

The present invention provides a drone inspection method, device, drone, and storage medium based on SwinTrack, the method comprising: obtaining a thermal image of the water surface during an inspection according to a water area inspection route; when a ship image to be detected is identified, a ship template image is matched from a preset thermal image template database according to the ship image to be detected; the ship template image and the ship image to be detected are input into a pre-trained SwinTrack model to generate a tracking prediction frame, and target tracking is performed according to the moving trajectory of the tracking prediction frame; when a tracking end signal is obtained and the current power level is greater than a preset power threshold, the inspection continues according to the water area inspection route. According to the technical solution of this embodiment, the SwinTrack model can be used to improve the accuracy of target tracking, so that the drone can quickly generate a tracking prediction frame for the ship, effectively improving the accuracy of target tracking of the ship at night and improving the reliability of water area management.

Description

Technical Field

[0001] The present invention belongs to the technical field of unmanned aerial vehicles, and in particular, relates to a unmanned aerial vehicle inspection method, a device, a unmanned aerial vehicle, and a storage medium based on SwinTrack. Background Art

[0002] Vessel monitoring is an important part of water area management. Due to the particularity of water area management, the traditional method can only rely on manual monitoring and surveillance on the water shoreline to monitor vessels, and the monitoring capacity is greatly limited. With the continuous improvement of drone hardware and software, drone inspection technology has emerged in the field of water area management. The flight characteristics of drones are used to enter the water area to track and locate ships, which greatly improves the reliability of water area management and expands the monitoring scope of water area management. However, there are no lighting facilities in the water area at night. Although infrared cameras can be used for night monitoring, due to the lack of image quality, it can only be used for stationary ship monitoring. It is difficult to achieve target tracking for moving ships, which affects the efficiency and reliability of water area management. Summary of the invention

[0003] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0004] The embodiments of the present invention provide a SwinTrack-based drone inspection method, device, drone, and storage medium, which can improve the tracking capability of the drone at night and improve the efficiency and reliability of water area management.

[0005] In a first aspect, an embodiment of the present invention provides a UAV inspection method based on SwinTrack, which is applied to a UAV, including:

[0006] Obtaining a pre-set water area inspection route, and obtaining a water surface thermal image during the inspection according to the water area inspection route;

[0007] Performing image recognition on the water surface thermal image, and when a ship image to be detected is recognized, matching a ship template image from a preset thermal image template database according to the ship image to be detected;

[0008] Inputting the ship template image and the ship image to be detected into a pre-trained SwinTrack model, generating a tracking prediction frame through the SwinTrack model, and performing target tracking according to the moving trajectory of the tracking prediction frame;

[0009] When the tracking end signal is obtained and the current power level is greater than the preset power level threshold, the inspection continues according to the water area inspection route.

[0010] In some embodiments, the step of inputting the vessel template image and the to-be-detected vessel image into a pre-trained SwinTrack model, and generating a tracking prediction frame through the SwinTrack model, comprises:

[0011] Segmenting the ship template image into a plurality of image blocks to obtain a first image block group;

[0012] Segmenting the to-be-detected ship image into a plurality of image blocks to obtain a second image block group;

[0013] Performing feature extraction and sequence conversion on the first image block group and the second image respectively to obtain a first feature sequence and a second feature sequence, wherein the first feature sequence is obtained based on the first image block group, and the second feature sequence is obtained based on the second image block group;

[0014] The first feature sequence and the second feature sequence are concatenated into a target feature sequence, and the target feature sequence is subjected to attention fusion to generate the tracking prediction frame.

[0015] In some embodiments, performing feature extraction and sequence conversion on the first image block group and the second image respectively to obtain a first feature sequence and a second feature sequence includes:

[0016] Performing feature extraction on the first image block group to obtain a first intermediate sequence, and determining a first sequence height and a first sequence width of the first intermediate sequence;

[0017] Performing feature extraction on the second image block group to obtain a second intermediate sequence, and determining a second sequence height and a second sequence width of the second intermediate sequence;

[0018] Get the preset hidden dimension and network stride;

[0019] converting the first intermediate sequence into the first feature sequence according to the hidden dimension, the network stride, the first sequence height, and the first sequence width;

[0020] The second intermediate sequence is converted into the second feature sequence according to the hidden dimension, the network stride, the second sequence height, and the second sequence width.

[0021] In some embodiments, the SwinTrack model includes an encoder and a decoder, and the generating the tracking prediction box after performing attention fusion on the target feature sequence includes:

[0022] Inputting the target feature sequence into the encoder, performing attention fusion through the encoder, and obtaining similarity information between the ship template image and the ship image to be detected;

[0023] Inputting the similarity information into the decoder, and obtaining a target feature map corresponding to the image of the ship to be detected through the decoder;

[0024] The tracking prediction box is generated according to the target feature map.

[0025] In some embodiments, generating the tracking prediction frame according to the target feature map includes:

[0026] Performing position encoding on the target feature map according to an unconstrained position encoding algorithm;

[0027] The loss of the target feature map after position encoding is calculated according to a preset loss function to obtain the tracking prediction box.

[0028] In some embodiments, the loss function includes Varifocal Loss and CIoU Loss.

[0029] In some embodiments, the drone is in communication with a management terminal, and the method further comprises:

[0030] Obtaining the water area inspection route sent by the management terminal;

[0031] When the ship template image is acquired, the ship reference information corresponding to the ship template image is fed back to the management terminal, the ship reference information including the ship type and the ship working parameters;

[0032] In the process of tracking the target according to the moving trajectory of the tracking prediction frame, sending the position information corresponding to the tracking prediction frame to the management terminal in real time;

[0033] When the tracking end signal sent by the management terminal is obtained and the current power level is greater than the power threshold, the inspection continues according to the water area inspection route.

[0034] In a second aspect, an embodiment of the present invention provides a drone inspection device based on SwinTrack, comprising:

[0035] An image acquisition unit, used to acquire a pre-set water area inspection route, and acquire a water surface thermal image during the inspection according to the water area inspection route;

[0036] An image recognition unit is used to perform image recognition on the water surface thermal image, and when a ship image to be detected is recognized, a ship template image is matched from a preset thermal image template database according to the ship image to be detected;

[0037] A target tracking unit, used for inputting the ship template image and the ship image to be detected into a pre-trained SwinTrack model, generating a tracking prediction frame through the SwinTrack model, and performing target tracking according to the moving trajectory of the tracking prediction frame;

[0038] The inspection recovery unit is used to continue to inspect according to the water area inspection route when a tracking end signal is obtained and the current power level is greater than a preset power threshold.

[0039] In a third aspect, an embodiment of the present invention provides a drone, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the drone inspection method based on SwinTrack as described in the first aspect is implemented.

[0040] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program is used to execute the SwinTrack-based drone inspection method as described in the first aspect.

[0041] The embodiment of the present invention includes: obtaining a pre-set water area inspection route, and obtaining a thermal image of the water surface during the inspection according to the water area inspection route; performing image recognition on the water surface thermal image, and when the image of the ship to be detected is identified, matching the ship template image from the preset thermal image template database according to the image of the ship to be detected; inputting the ship template image and the image of the ship to be detected into a pre-trained SwinTrack model, generating a tracking prediction frame through the SwinTrack model, and tracking the target according to the moving trajectory of the tracking prediction frame; when the tracking end signal is obtained and the current power is greater than the preset power threshold, continue to inspect according to the water area inspection route. According to the technical solution of this embodiment, the SwinTrack model can be used to improve the accuracy of target tracking based on the attention mechanism of deep learning, so that the drone can quickly generate a tracking prediction frame of the ship from the captured thermal image, effectively improving the accuracy of target tracking of the ship at night and improving the reliability of water area management.

[0042] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings are used to provide a further understanding of the technical solution of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the technical solution of the present invention and do not constitute a limitation on the technical solution of the present invention.

[0044] Figure 1 is a flow chart of a SwinTrack-based drone inspection method provided by an embodiment of the present invention;

[0045] Figure 2 is a flowchart of segmenting an image provided by another embodiment of the present invention;

[0046] Figure 3 is a flow chart of feature extraction and feature conversion provided by another embodiment of the present invention;

[0047] Figure 4 is a flowchart of encoding and decoding provided by another embodiment of the present invention;

[0048] Figure 5 is a network diagram of a SwinTrack model provided by another embodiment of the present invention;

[0049] Figure 6 is a flowchart of generating a tracking prediction frame provided by another embodiment of the present invention;

[0050] Figure 7 is a flow chart of a management terminal controlling a drone provided by another embodiment of the present invention;

[0051] Figure 8 is a structural diagram of a SwinTrack-based drone inspection device provided by another embodiment of the present invention;

[0052] Fig. 9 2 is a device diagram of a drone provided by another embodiment of the present invention. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0054] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "target" and the like in the specification, claims or the above drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0055] The present invention provides a method, device, drone and storage medium for drone inspection based on SwinTrack, the method comprising: obtaining a pre-set water area inspection route, obtaining a thermal image of the water surface during the inspection according to the water area inspection route; performing image recognition on the thermal image of the water surface, and when the image of the ship to be detected is identified, matching a ship template image from a preset thermal image template database according to the image of the ship to be detected; inputting the ship template image and the image of the ship to be detected into a pre-trained SwinTrack model, generating a tracking prediction frame through the SwinTrack model, and tracking the target according to the moving trajectory of the tracking prediction frame; when a tracking end signal is obtained and the current power is greater than a preset power threshold, continuing to inspect according to the water area inspection route. According to the technical solution of this embodiment, the SwinTrack model can be used to improve the accuracy of target tracking based on the attention mechanism of deep learning, so that the drone can quickly generate a tracking prediction frame of the ship from the captured thermal image, effectively improving the accuracy of target tracking of the ship at night and improving the reliability of water area management.

[0056] like Figure 1 As shown, Figure 1 The flowchart of a SwinTrack-based UAV inspection method provided by an embodiment of the present invention is applied to a UAV, including but not limited to the following steps:

[0057] Step S110, obtaining a pre-set water area inspection route, and obtaining a water surface thermal image during the inspection according to the water area inspection route;

[0058] Step S120, performing image recognition on the water surface thermal image, and when a ship image to be detected is recognized, matching a ship template image from a preset thermal image template database according to the ship image to be detected;

[0059] Step S130, inputting the ship template image and the image of the ship to be detected into a pre-trained SwinTrack model, generating a tracking prediction frame through the SwinTrack model, and tracking the target according to the moving trajectory of the tracking prediction frame;

[0060] Step S140, when the tracking end signal is obtained and the current power level is greater than the preset power level threshold, the inspection continues according to the water area inspection route.

[0061] It should be noted that the water area inspection route can be obtained in any way, such as sending it to the drone through the management terminal, so as to achieve flexible adjustment of the water area inspection route. If there is a fixed route in the water area, the water area inspection route can also be pre-set in the drone, so as to improve the inspection efficiency of the drone. It is worth noting that when there is an inspection route, the process of controlling the drone for inspection is a technology well known to those skilled in the art, and will not be elaborated here.

[0062] It is worth noting that ordinary optical cameras have low visibility and clarity at night. In order to realize night inspections, thermal imaging cameras can be used as the cameras of drones. The thermal imaging cameras can be used to photograph the water surface during the inspection process to obtain thermal images of the water surface, effectively improving the clarity and accuracy of images obtained by drones during night inspections, and providing an image basis for subsequent image recognition.

[0063] It should be noted that since the engine of a ship generates heat during operation, and the engines of different ships are usually different, the thermal images of the engines of different ships can be collected in advance as reference images and saved in a database as a thermal image template database. After the image of the ship to be detected is identified from the water surface thermal image, the corresponding part is intercepted to match the ship template image from the thermal image template database, for example Figure 5 As shown in the leftmost figure below, when a ship is sailing at night, the main source of heat is the ship engine. Therefore, after obtaining the water surface thermal map, the part with a thermal value greater than a preset threshold can be used as the ship image to be detected. The ship image to be detected is used for image matching in the thermal image template database, and the template image with the highest similarity is used as the ship template image.

[0064] It should be noted that the SwinTrack model is a target tracking framework based on the deep learning attention mechanism. It needs to input the template image and the image to be detected at the same time. In this embodiment, the ship template image is used as the template image, and the image of the ship to be detected is used as the image to be detected. The SwinTrack model is used to determine the similarity between the image to be detected and the template image, thereby generating a tracking prediction box to achieve target tracking. The specific model structure of the SwinTrack model can be determined according to actual needs. For the sake of simplicity, this embodiment uses Figure 5 The network structure shown is used as an example for illustration. The SwinTrack model includes five parts, namely, a network backbone (Backbone), an encoder (Encoder), a decoder (Decoder), a positional encoding (PositionalEncoding), and a head network (Head). Those skilled in the art can also adjust the network according to actual needs, which is not a limitation of this embodiment.

[0065] It should be noted that after tracking a ship in the waters, when the tracking requirements are met, the drone can be controlled to stop target tracking, such as tracking for a period of time, or sending a tracking stop signal through the management terminal. After stopping tracking, in order to ensure the safety of the drone, it can be determined whether to continue the inspection based on the power level. When the power level is greater than the preset threshold, it can be determined that the drone has at least enough power to return to the charging device to continue the inspection. When the power level drops to the preset threshold, the drone needs to return to the charging device for charging. The specific method of calculating the power level and return path is a technology well known to technicians in this field and will not be elaborated here.

[0066] It should be noted that in order to improve the energy replenishment efficiency of the UAV, a battery replacement device can be used to replace the battery of the UAV. The battery replacement device can be composed of three modules: a robotic arm responsible for loading and unloading UAV batteries, multiple UAV battery charging compartments, and a small airport for UAV take-off and landing and standby. The specific structure can be adjusted according to actual needs and is not limited here.

[0067] In addition, in one embodiment, referring to Figure 2 , Figure 1 Step S130 of the illustrated embodiment also includes but is not limited to the following steps:

[0068] Step S210, dividing the ship template image into a plurality of image blocks to obtain a first image block group;

[0069] Step S220, dividing the image of the ship to be detected into a plurality of image blocks to obtain a second image block group;

[0070] Step S230, performing feature extraction and sequence conversion on the first image block group and the second image, respectively, to obtain a first feature sequence and a second feature sequence, wherein the first feature sequence is obtained based on the first image block group, and the second feature sequence is obtained based on the second image block group;

[0071] Step S240: concatenate the first feature sequence and the second feature sequence into a target feature sequence, and generate a tracking prediction frame after performing attention fusion on the target feature sequence.

[0072] It should be noted that image segmentation can be achieved through common data preprocessing methods, and the image can be divided into multiple small blocks. By inputting the second feature sequence and the first feature sequence into the SwinTrack model, the SwinTrack model can be learned according to the first feature sequence, so that the extracted features can be more focused on the ship itself.

[0073] It should be noted that the ship template image and the ship image to be detected can be divided into blocks, feature extraction and sequence conversion can be performed in the network skeleton of the SwinTrack model, and splicing can be achieved in the spatial dimension by weight sharing in the encoder to obtain the target feature sequence, which is sequentially input into the decoder, the position encoding module and the head network to obtain the tracking prediction box. This embodiment does not limit the specific attention fusion process.

[0074] In addition, in one embodiment, referring to Figure 3 , Figure 2 Step S230 of the illustrated embodiment also includes but is not limited to the following steps:

[0075] Step S310, extracting features from the first image block group to obtain a first intermediate sequence, and determining a first sequence height and a first sequence width of the first intermediate sequence;

[0076] Step S320, extracting features from the second image block group to obtain a second intermediate sequence, and determining a second sequence height and a second sequence width of the second intermediate sequence;

[0077] Step S330, obtaining a preset hidden dimension and network stride;

[0078] Step S340, converting the first intermediate sequence into a first feature sequence according to the hidden dimension, the network stride, the first sequence height and the first sequence width;

[0079] Step S350, converting the second intermediate sequence into a second feature sequence according to the hidden dimension, the network stride, the second sequence height and the second sequence width.

[0080] It should be noted that the ship template image is used as The image of the ship to be detected is As an example, the multi-head attention mechanism is used to process the divided image, and the first intermediate sequence and the second intermediate sequence are obtained respectively. and Among them, H z , H x are the height of the feature sequence of the ship template image and the image to be detected, W z , W x are the widths of the feature sequences of the ship template image and the image to be detected, CC is the hidden dimension of the entire model, and S is the stride of the network skeleton. Through the network skeleton, features can be extracted from the two sets of images, making the model more focused on the features of the airship itself.

[0081] In addition, in one embodiment, the SwinTrack model includes an encoder and a decoder. Figure 4 , Figure 2Step S240 of the illustrated embodiment also includes but is not limited to the following steps:

[0082] Step S410, inputting the target feature sequence into the encoder, performing attention fusion through the encoder, and obtaining similarity information between the ship template image and the ship image to be detected;

[0083] Step S420, inputting the similarity information into a decoder, and obtaining a target feature map corresponding to the image of the ship to be detected through the decoder;

[0084] Step S430, generating a tracking prediction box according to the target feature map.

[0085] It should be noted that in Figure 5 In the structure shown, the encoder consists of N blocks, each of which contains a multi-headed self-attention (MSA) module and a feedforward network (FFN). Among them, the FFN contains two layers of perceptron (Multilayer Perceptron, MLP), and a GELU activation layer is added after the output of the first layer of perceptron. In addition, layer normalization (LN) is added before each MSA and FFN.

[0086] It is worth noting that before the first feature sequence and the second feature sequence are input into the encoder, the two feature sequences can be spliced ​​into a unified sequence U according to the spatial dimension, and the sequence U is used as the input of the encoder, and then corrected by the FFN after passing through the self-attention mechanism of MSA. When the encoder outputs, the sequence U will be split back into the feature sequences of the template image and the thermal image. The specific process is as follows:

[0087]

[0088] z L ,x L =DeConcat(U L ), where l is the lth layer, L is the number of image blocks, z 1 ,x 1 is the feature sequence of the corresponding ship template image and the image to be detected generated by the network skeleton, z L ,x L is the output of the encoder. The similarity between the airship template image and the airship image to be detected can be learned through the multi-head attention mechanism of the Encoder.

[0089] The decoder consists of a multi-head crisscross attention (MCA) module and a feed-forward network (FFN). The output of the encoder is used as the input of the decoder by calculating x Land Concat(z L ,x L ) to obtain the target feature map of the airship image to be detected The specific process is as follows:

[0090] U D =Concat(z L ,x L ),

[0091] x L '=x L +MCA(LN(x L ),LN(U D )),

[0092] x=x L '+FFN(LN(x L ')).

[0093] In addition, in one embodiment, referring to Figure 6 , Figure 4 Step S430 of the illustrated embodiment also includes but is not limited to the following steps:

[0094] Step S610, position encoding the target feature map according to an unconstrained position encoding algorithm;

[0095] Step S620, performing loss calculation on the position-encoded target feature map according to a preset loss function to obtain a tracking prediction box.

[0096] It should be noted that since the SwinTrack model requires position encoding to identify the position of the currently processed token, this embodiment selects the joint position encoding of the unconstrained position encoding algorithm as the position encoding method of SwinTrack, adds a learnable position encoding p to the self-attention module, and adds a relative position deviation b i-j As a supplement to the absolute position encoding. Finally, in order to achieve splicing-based fusion, the unconstrained absolute position encoding is spliced ​​to match the actual position, and a pair of indexes g, h are added to the relative position deviation index tuple to reflect the current query (query, Q) and key (key, K), resulting in the following equation:

[0097]

[0098] Taking the two-dimensional feature sequence, at the encoder layer l as an example, we have:

[0099] Where x is each token in the feature sequence; i, j, m, n represent the coordinates of token x in the feature sequence; g, h represent the tokens from the template image and the image to be detected before concatenation; U is the learnable projection matrix of the linear transformation of p; d is the dimension of the key (key, K).

[0100] In addition, Figure 5 In the structure shown, the head network is divided into two branches: the classification branch and the bounding box (BBOX) regression branch. Each branch has three layers of perceptrons, and both branches take the output of the decoder as input. After the decoder output sequence of the target feature sequence passes through the head network, the SwinTrack model will mark the current location of the ship in the image of the ship to be detected in the form of a prediction box, thereby realizing target tracking for the ship.

[0101] In addition, in one embodiment, the loss function includes Varifocal Loss and CIoU Loss.

[0102] It should be noted that in order to eliminate the gap between different prediction branches, the ground-truth value can be replaced with the IoU between the predicted BBox and the ground-truth, i.e., IACS, during classification. IACS can help the model select a more accurate bounding box from the candidates. Based on this, the Varifocal loss of this embodiment has the following form:

[0103] Where p is the IACS score and q is the preset target score. For positive samples, i.e. foreground points, q is the IoU between the predicted bounding box and the ground-truth bounding box. For negative samples, q = 0, then the classification loss can be expressed as: Where b is the predicted bounding box, is the ground-truth bounding box.

[0104] It should be noted that for Figure 5 For the BBOX regression shown, you can choose the general IoU loss. The specific function formula is as follows: The CIoU loss is weighted by p, emphasizing the high classification scores of samples, while the training signals of negative samples are ignored.

[0105] In addition, in one embodiment, the drone is connected to the management terminal for communication. Figure 7 The method of this embodiment also includes but is not limited to the following steps:

[0106] Step S710, obtaining the water area inspection route sent by the management terminal;

[0107] Step S720: when the ship template image is acquired, the ship reference information corresponding to the ship template image is fed back to the management terminal, where the ship reference information includes the ship type and the ship working parameters;

[0108] Step S730, in the process of tracking the target according to the moving trajectory of the tracking prediction frame, sending the position information corresponding to the tracking prediction frame to the management terminal in real time;

[0109] Step S740, when the tracking end signal sent by the management terminal is obtained and the current power level is greater than the power threshold, the inspection continues according to the water area inspection route.

[0110] It should be noted that the management terminal can be a common mobile phone APP and / or a web-based management platform. The specific type of management terminal can be selected according to actual needs. Of course, for the convenience of management, both the web-based management platform and the mobile phone app can be used at the same time. For example, the inspection route of the drone can be formulated and task instructions can be issued through the web-based management platform, so as to realize the simultaneous management of multiple drones to perform multiple groups of inspection tasks, real-time monitoring of drone inspection images, and viewing of drone inspection process playback, etc., which can efficiently cooperate with ground command and dispatch work; view the status and information of the drone when performing the task through the mobile phone APP, turn on manual / automatic flight mode to meet the needs of different scenarios, and it also has real-time inspection live broadcast, photo taking, video recording and other functions. Technical personnel in this field are motivated to adjust the functions that can be achieved by the management terminal according to actual needs, and will not go into details here.

[0111] It should be noted that when a ship template image is obtained, the specific type of ship to be tracked can be determined, and the ship reference information can be fed back to the management terminal so that the management personnel can make different monitoring adjustments according to different ship types and working parameters.

[0112] It should be noted that during the target tracking process of the UAV, the monitoring video and location information of the target ship can be fed back to the management terminal in real time, thereby improving the monitoring effect.

[0113] In addition, refer to Figure 8 The embodiment of the present invention provides a SwinTrack-based drone inspection device, the SwinTrack-based drone inspection device 800 includes

[0114] The image acquisition unit 810 is used to acquire a preset water area inspection route and acquire a water surface thermal image during the inspection according to the water area inspection route;

[0115] The image recognition unit 820 is used to perform image recognition on the water surface thermal image, and when a ship image to be detected is recognized, a ship template image is matched from a preset thermal image template database according to the ship image to be detected;

[0116] The target tracking unit 830 is used to input the ship template image and the image of the ship to be detected into the pre-trained SwinTrack model, generate a tracking prediction frame through the SwinTrack model, and track the target according to the moving trajectory of the tracking prediction frame;

[0117] The inspection recovery unit 840 is used to continue the inspection according to the water area inspection route when the tracking end signal is obtained and the current power level is greater than the preset power threshold.

[0118] In addition, refer to Fig. 9 An embodiment of the present invention further provides a drone, the drone 900 comprising: a memory 910, a processor 920, and a computer program stored in the memory 910 and executable on the processor 920.

[0119] The processor 920 and the memory 910 may be connected via a bus or in other ways.

[0120] The non-transient software program and instructions required to implement the SwinTrack-based drone inspection method of the above embodiment are stored in the memory 910. When executed by the processor 920, the SwinTrack-based drone inspection method of the above embodiment is executed, for example, the above-described Figure 1 Steps S110 to S140 of the method, Figure 2 Steps S210 to S240 of the method, Figure 3 Steps S310 to S350 of the method, Figure 4 Steps S410 to S430 of the method, Figure 6 Steps S610 to S620 of the method, Figure 7 The method comprises steps S710 to S740.

[0121] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0122] In addition, one embodiment of the present invention further provides a computer-readable storage medium, which stores a computer program. The computer program is executed by a processor or a controller, for example, by a processor in the above-mentioned drone embodiment, so that the above-mentioned processor can execute the drone inspection method based on SwinTrack in the above-mentioned embodiment, for example, execute the above-mentioned Figure 1 Steps S110 to S140 of the method, Figure 2 Steps S210 to S240 of the method, Figure 3 Steps S310 to S350 of the method, Figure 4 Steps S410 to S430 of the method, Figure 6 Steps S610 to S620 of the method, Figure 7 Method steps S710 to S740 in the method. It will be appreciated by those skilled in the art that all or some of the steps and devices in the method disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. Some physical components or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable storage medium, which may include a computer storage medium (or a non-transitory storage medium) and a communication storage medium (or a temporary storage medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable storage media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other storage medium that may be used to store desired information and may be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication storage media generally contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery storage media.

[0123] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the embodiments disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application.

[0124] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

[0125] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the above-mentioned implementation mode. Technical personnel familiar with the field can also make various equivalent deformations or substitutions without violating the spirit of the present invention. These equivalent deformations or substitutions are all included in the scope defined by the claims of the present invention.

Claims

1. A UAV inspection method based on SwinTrack, characterized in that: Applications in drones include: Obtaining a pre-set water area inspection route, and obtaining a water surface thermal image during the inspection according to the water area inspection route; Performing image recognition on the water surface thermal image, and when a ship image to be detected is recognized, matching a ship template image from a preset thermal image template database according to the ship image to be detected; Inputting the ship template image and the ship image to be detected into a pre-trained SwinTrack model, generating a tracking prediction frame through the SwinTrack model, and performing target tracking according to the moving trajectory of the tracking prediction frame; When the tracking end signal is obtained and the current power level is greater than the preset power level threshold, the inspection continues according to the water area inspection route.

2. The SwinTrack-based drone inspection method according to claim 1, characterized in that: The step of inputting the vessel template image and the to-be-detected vessel image into a pre-trained SwinTrack model and generating a tracking prediction frame through the SwinTrack model includes: Segmenting the ship template image into a plurality of image blocks to obtain a first image block group; Segmenting the to-be-detected ship image into a plurality of image blocks to obtain a second image block group; Performing feature extraction and sequence conversion on the first image block group and the second image respectively to obtain a first feature sequence and a second feature sequence, wherein the first feature sequence is obtained based on the first image block group, and the second feature sequence is obtained based on the second image block group; The first feature sequence and the second feature sequence are concatenated into a target feature sequence, and the target feature sequence is subjected to attention fusion to generate the tracking prediction frame.

3. The SwinTrack-based drone inspection method according to claim 2, characterized in that: The first image block group and the second image are subjected to feature extraction and sequence conversion respectively to obtain a first feature sequence and a second feature sequence, including: Performing feature extraction on the first image block group to obtain a first intermediate sequence, and determining a first sequence height and a first sequence width of the first intermediate sequence; Performing feature extraction on the second image block group to obtain a second intermediate sequence, and determining a second sequence height and a second sequence width of the second intermediate sequence; Get the preset hidden dimension and network stride; converting the first intermediate sequence into the first feature sequence according to the hidden dimension, the network stride, the first sequence height, and the first sequence width; The second intermediate sequence is converted into the second feature sequence according to the hidden dimension, the network stride, the second sequence height, and the second sequence width.

4. The SwinTrack-based drone inspection method according to claim 2, characterized in that: The SwinTrack model includes an encoder and a decoder, and the generating of the tracking prediction frame after attention fusion of the target feature sequence includes: Inputting the target feature sequence into the encoder, performing attention fusion through the encoder, and obtaining similarity information between the ship template image and the ship image to be detected; Inputting the similarity information into the decoder, and obtaining a target feature map corresponding to the image of the ship to be detected through the decoder; The tracking prediction box is generated according to the target feature map.

5. The SwinTrack-based UAV inspection method according to claim 4, characterized in that: The generating the tracking prediction frame according to the target feature map includes: Performing position encoding on the target feature map according to an unconstrained position encoding algorithm; The loss of the target feature map after position encoding is calculated according to a preset loss function to obtain the tracking prediction box.

6. The SwinTrack-based UAV inspection method according to claim 5, characterized in that: The loss functions include Varifocal Loss and CIoU Loss.

7. The SwinTrack-based drone inspection method according to claim 1, characterized in that: The drone is in communication connection with the management terminal, and the method further comprises: Obtaining the water area inspection route sent by the management terminal; When the ship template image is acquired, the ship reference information corresponding to the ship template image is fed back to the management terminal, the ship reference information including the ship type and the ship working parameters; In the process of tracking the target according to the moving trajectory of the tracking prediction frame, sending the position information corresponding to the tracking prediction frame to the management terminal in real time; When the tracking end signal sent by the management terminal is obtained and the current power level is greater than the power threshold, the inspection continues according to the water area inspection route.

8. A drone inspection device based on SwinTrack, characterized in that: include: An image acquisition unit, used to acquire a pre-set water area inspection route, and acquire a water surface thermal image during the inspection according to the water area inspection route; An image recognition unit is used to perform image recognition on the water surface thermal image, and when a ship image to be detected is recognized, a ship template image is matched from a preset thermal image template database according to the ship image to be detected; A target tracking unit, used for inputting the ship template image and the ship image to be detected into a pre-trained SwinTrack model, generating a tracking prediction frame through the SwinTrack model, and performing target tracking according to the moving trajectory of the tracking prediction frame; The inspection recovery unit is used to continue to inspect according to the water area inspection route when a tracking end signal is obtained and the current power level is greater than a preset power threshold.

9. A drone, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the SwinTrack-based drone inspection method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: The computer program is used to execute the SwinTrack-based drone inspection method as described in any one of claims 1 to 7.

Images