Vehicle-mounted unmanned aerial vehicle accompanying flight control method and device and computer readable storage medium
Through the target image taken by the camera on the drone, the target detection model is used to identify and repair the characteristics of the accompanying target object, the problem of inaccurate positioning of the vehicle-mounted drone in the environment of disturbed GPS signal is solved, and the precise accompanying flight of the drone in different scenarios is achieved.
Patent Information
- Application Number
- CN202510314400.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-17
- Publication Date
- 2025-06-24
AI Technical Summary
The existing vehicle-mounted drone companion flight methods rely on network data or specific signal transmission systems, resulting in interference of GPS signals in dense urban high-rise areas, tunnels or forests, affecting the positioning accuracy of the drone.
The target image taken by the camera on the drone is identified using the target detection model, and the accompanying target object is determined, and feature repair is carried out when the target object is partially blocked to ensure that the drone can accurately follow the vehicle.
It realizes that drones can fly accurately without relying on external network data in different scenarios, solves the problem of inaccurate positioning caused by interference from GPS signals, and improves detection accuracy.
Smart Images
Figure CN120198645A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the technical field of vehicle-mounted drones, and particularly to a method, device, and computer-readable storage medium for controlling the accompanying flight of vehicle-mounted drones. Background Art
[0002] With the rapid development of technology, vehicle-mounted drones, as a new type of auxiliary tool, are increasingly widely used in fields such as vehicle monitoring and aerial photography. More and more people choose to use drones for travel photography when driving. However, most of the existing methods for controlling the accompanying flight of vehicle-mounted drones rely on network data or specific signal transmission systems, which limits their application scenarios.
[0003] Generally, vehicle-mounted drones use the satellite positioning system for accompanying flight. When in an open area, GPS can provide very accurate position information for the drone, ensuring that it can closely follow the vehicle's movement. However, in places with high-rise buildings in the city, inside tunnels, or in dense forests, the GPS signal is easily blocked or interfered with, affecting the positioning accuracy of the drone. Therefore
[0004] In open areas, GPS can provide high-precision positioning information, enabling the drone to accurately follow the vehicle. But in urban high-rise dense areas, inside tunnels, or in forests, the GPS signal may be interfered with, affecting the drone's positioning and resulting in inaccurate positioning problems. Summary of the Invention
[0005] To overcome the problems existing in the related art, this specification provides a method, device, and computer-readable storage medium for controlling the accompanying flight of vehicle-mounted drones.
[0006] According to the first aspect of the embodiments of this specification, a method for controlling the accompanying flight of vehicle-mounted drones is provided. The method includes:
[0007] Obtain a target image captured by a camera on the drone;
[0008] Identify the accompanying target object by using a target detection model on the target image. Wherein, when it is detected that the accompanying target object is partially occluded, the target detection model is used to repair the features of the accompanying target object;
[0009] According to the position information of the accompanying target object in the target image after repair, determine the azimuth information of the accompanying target object relative to the drone, and control the drone to continuously accompany the accompanying target object.
[0010] According to a vehicle-mounted UAV accompanying flight control method provided by the present application, the target detection model includes a Backbone module, a feature repair module, a Neck module, and a detection module; the feature repair module is connected between the Backbone module and the Neck module;
[0011] The Backbone module is used to extract image features of the target image;
[0012] The feature repair module is used to repair the image features in the case of image feature loss, where the case of image feature loss includes that the accompanying target object is partially occluded;
[0013] The Neck module is used to perform different-scale processing on the repaired image features, extract different-scale features of the repaired image features, and perform multi-scale fusion on the different-scale features of the repaired image features;
[0014] The detection module is used to identify the target image based on the fused feature map to obtain the complete accompanying target object after repair.
[0015] According to a vehicle-mounted UAV accompanying flight control method provided by the present application, the feature repair module repairs the image features based on a feature repair network according to a feature repair method in the case of image feature loss;
[0016] Among them, the feature repair method includes:
[0017] Generating new feature values for the occluded part of the accompanying target object to obtain a first repaired image;
[0018] Performing local discrimination on the first image to obtain a first output result;
[0019] Performing global discrimination on the target image where the first repaired image is located to obtain a second output result;
[0020] According to the first output result and the second output result, the repaired image features are obtained.
[0021] According to a vehicle-mounted UAV accompanying flight control method provided by the present application, the feature repair module includes a feature repair network based on a generative adversarial network, and the feature repair network includes a generator, a local discriminator, and a global discriminator;
[0022] The generator generates new feature values for the occluded part;
[0023] The local discriminator compares the difference between the occluded part and the real part of the new feature values;
[0024] Compare the differences between the entire image after feature repair and the real identification objects in the real image through the global discriminator;
[0025] Iterate the generator and the two discriminators until the two discriminators cannot determine whether the generated new feature values are the features of the real identification objects, and obtain the repaired image features.
[0026] According to a vehicle-mounted UAV accompanying flight control method provided by the present application, the target detection model further includes a shallow layer module; the shallow layer module is connected to the shallow Backbone module;
[0027] The method further includes:
[0028] Extract features from the image features obtained after being processed by the Backbone module to obtain a shallow feature map;
[0029] Based on the shallow feature map, detect the accompanying target objects in the target image with sizes smaller than the threshold.
[0030] According to a vehicle-mounted UAV accompanying flight control method provided by the present application, the shallow layer module is connected after the deep feature repair module, and the shallow layer module is connected to the Neck module;
[0031] The method further includes:
[0032] The image features repaired by the feature repair module are input into the Neck module;
[0033] The shallow feature map extracted by the shallow layer module is input into the Neck module;
[0034] Fuse the repaired image features and the shallow feature map through the Neck module to detect the accompanying target objects in the target image.
[0035] According to a vehicle-mounted UAV accompanying flight control method provided by the present application, the method further includes:
[0036] The accompanying target objects have multiple nested icons, and among them, the sizes of the multiple icons are set step by step according to the recognition accuracy of the target detection model.
[0037] According to a vehicle-mounted UAV accompanying flight control method provided by the present application, the method further includes:
[0038] The target detection model includes a yoloN network model; among them, the yoloN network model includes a yolo11n model.
[0039] According to a vehicle-mounted UAV accompanying flight control method provided by the present application, the method further includes:
[0040] When the accompanying target object in the target image is completely blocked, obtain the GPS signal of the accompanying target object;
[0041] Determine the azimuth information of the accompanying target object relative to the UAV according to the GPS signal, and control the UAV to continuously accompany the accompanying target object.
[0042] The present application also provides a vehicle-mounted UAV accompanying flight control device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the vehicle-mounted UAV accompanying flight control method as described in any one of the above.
[0043] The present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the vehicle-mounted UAV accompanying flight control method as described in any one of the above.
[0044] In the embodiments of this specification, for the vehicle-mounted UAV accompanying flight control method, device, and computer-readable storage medium, compared with the current implementation of UAV accompanying flight through satellite positioning technology, where the GPS signal is easily interfered with and the positioning is inaccurate in the city, obtain the target image captured by the camera on the UAV; identify the accompanying target object by using a target detection model for the target image. Through visual recognition, it is possible to achieve precise accompanying flight of the UAV in different scenarios without relying on external network data, and solve the problems that occur when accompanying flight is based on the GPS signal. Moreover, when it is detected that the accompanying target object is partially blocked, the target detection model is used to repair the features of the accompanying target object to improve the detection accuracy of the detection network for the accompanying target object. When the target detection model detects the repaired accompanying target object, determine the azimuth information of the accompanying target object relative to the UAV according to the position information of the repaired accompanying target object in the target image, and control the UAV to continuously accompany the accompanying target object.
[0045] It should be understood that the above general description and subsequent detailed description are only exemplary and explanatory, and cannot limit this specification. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] The accompanying drawings here are incorporated into the specification and constitute a part of this specification, showing embodiments consistent with this specification, and are used together with the specification to explain the principles of this specification.
[0047] Figure 1 It is a schematic flowchart of the vehicle-mounted UAV accompanying flight control method shown according to an exemplary embodiment of this specification;
[0048] Figure 2 It is a schematic diagram of the icon of the accompanying target object shown in this specification according to an exemplary embodiment;
[0049] Figure 3 It is the overall structure diagram of the target detection model shown in this specification according to an exemplary embodiment;
[0050] Figure 4 It is the GAN-FRM network structure diagram shown in this specification according to an exemplary embodiment;
[0051] Figure 5 It is a schematic diagram of a vehicle-mounted UAV accompanying flight control device shown in this specification according to an exemplary embodiment;
[0052] Figure 6 It is a schematic block diagram of a vehicle-mounted UAV accompanying flight control device shown in this specification according to an exemplary embodiment. Detailed implementation manners
[0053] Here, the technical solutions in the embodiments (or "implementation manners") of the present application will be clearly and completely described in conjunction with the accompanying drawings. When the following description involves the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements.
[0054] If there are terms related to directional indications or positional relationships in the embodiments of the present application (such as up, down, left, right, front, back, inside, outside, top, bottom, center, vertical, horizontal, longitudinal, transverse, length, width, counterclockwise, clockwise, axial, radial, circumferential, etc.), such terms are only used to explain the relative positional relationships and movement conditions between components in a specific posture (as shown in the accompanying drawings); if this specific posture changes, then the directional indication or positional relationship will also change accordingly. In addition, the terms "first", "second", etc. involved in the embodiments of the present application are only for the purpose of convenient description and cannot be understood as indicating or implying relative importance.
[0055] The present application provides a vehicle-mounted UAV accompanying flight control method, device, and computer-readable storage medium. The present application will be described in detail below in conjunction with the accompanying drawings. Without conflict, the features in the following embodiments and implementation manners can be combined with each other.
[0056] On-vehicle drones, as a new type of auxiliary tool, are increasingly widely used in fields such as vehicle monitoring and aerial photography. In most cases, on-vehicle drones use the satellite positioning system for accompanying flight. When in an open area, GPS can provide very accurate position information for the drone, ensuring that it can closely follow the vehicle's movement. However, in areas with numerous high-rise buildings in the city, inside tunnels, or in dense forests, due to possible obstruction or interference of GPS signals, the positioning accuracy of the drone is affected.
[0057] To solve the above technical problems, this specification provides an on-vehicle drone accompanying flight control method.
[0058] It realizes accompanying flight through visual algorithms, aiming to achieve precise accompanying flight control in scenarios where external network data is not relied on and GPS signals are interfered with.
[0059] It should be noted that in the field of computer vision, there are mainly two types of methods: traditional vision-based and deep learning-based. Among them, traditional vision methods are mostly applicable to scenarios where the camera is stationary and lack practicality in environments with variable backgrounds such as those where drones are located. Although deep learning methods have a wide range of application scenarios and strong anti-interference capabilities, they have problems such as large computational amounts and the need for sufficient computing power support, and their effects are not good when encountering branch occlusion during vehicle driving and detecting small targets at relatively high altitudes. This case focuses on the difficulties of deep learning visual algorithms, implements optimization measures from the overall detection process and methods, and focuses on improving and perfecting the problems existing in the model in aspects such as small target detection, occlusion processing, and computational amount optimization, aiming to improve its performance and application effects.
[0060] Figure 1 It is a schematic flowchart of an on-vehicle drone accompanying flight control method provided by an embodiment of this specification, including the following steps:
[0061] Step 100, obtain the target image captured by the camera on the drone.
[0062] It should be noted that the on-vehicle drone accompanying flight control method in the text is applied to any device or system such as a terminal, a vehicle, an independent server, a server cluster, the cloud, a vehicle network platform, an Internet of Things platform, etc.
[0063] The camera mounted on the drone captures the target image of its surrounding environment in real time and transmits the target image to the embedded processing system on the vehicle or the drone for subsequent target detection and analysis.
[0064] Under normal circumstances, the target image includes a companion target object, and subsequently the drone needs to use the companion target object as a reference for following. The companion target object described in this article refers to an object that the drone needs to track and maintain a relatively stable position with during the drone following mission scenario. The companion target object has recognizable features of Indigo Plaza, which can be but are not limited to visual features such as appearance shape, color, texture, etc., or have specific markings and identifiers. Exemplarily, the companion target object is a specific identifier, a vehicle carrying the drone, a certain component / area on the vehicle, etc. Preferably, a specific identifier is used as the companion target object.
[0065] As an example, the companion target object is an icon with identification features. The shape, color, size, etc. of the icon are not limited and are set according to actual needs. Its shape can be but is not limited to square, circular, etc. Place the completed companion target object on the vehicle carrying the drone, such as on the roof, vehicle hood, etc., where there is an obvious line of sight.
[0066] As another example, the companion target object has multiple nested icons, where the sizes of the multiple icons are set step by step according to the recognition accuracy of the target detection model.
[0067] The multi-layer nested structure can provide multiple recognition levels under different observation conditions of the drone. At the same time, for easy distinction, more prominent colors are more likely to be detected by the model. Therefore, the colors of each icon are different to improve the detection accuracy.
[0068] It can be understood that when the drone is at a relatively high flight altitude, it may only be able to recognize the outermost layer; while when the distance is closer and the flight altitude is lower, the inner layer can be recognized, which can adapt to different flight scenarios and target distances and increase the chance of the target being successfully recognized. Refer to Figure 2 , when the drone is at a relatively high flight altitude, the middle area may not be clearly visible. Taking the outer frame area as the standard, only the outer area needs to be detected. When the drone is at a relatively low flight altitude, the outer frame area may not appear completely in the image. When the inner identifier can be recognized, positioning with the inner identifier is more accurate. The middle area is between the two. In short, take the smallest identifier recognized by the drone as the target.
[0069] In summary, the use of the three areas can complement each other. When a certain area is missed, as long as one of the areas is detected. At the same time, from the perspective of information redundancy, the multi-layer structure provides more detailed information about the target, and these information can complement each other, which helps to more accurately identify the target.
[0070] Step 200, identify the accompanying target object by using the target detection model for the target image; wherein, when it is detected that the accompanying target object is partially occluded, the target detection model is used to repair the features of the accompanying target object.
[0071] The target detection model uses deep learning algorithms (such as convolutional neural networks) to extract and analyze the features of the input target image. It searches for regions in the target image that match the predefined features of the accompanying target object. For example, if the accompanying target object is a specific icon, the model will look for objects with similar features in the target image based on the shape, color, texture and other feature information of the icon learned in advance.
[0072] As an example, the target detection model includes the yoloN network model; wherein, the yoloN network model includes the yolo11n model.
[0073] The deep learning-based method has a wide range of applicable scenarios and strong anti-interference ability, but it has a large amount of computation and requires powerful computing resources as support. To optimize the problem of the amount of computation, this paper proposes a target detection model improved based on yolo11n. Yolo11 is the latest model in the yolo series, which has been greatly optimized for the amount of computation and is extremely efficient in terms of processing ability. In this case, the yolo11n network is improved based on the smallest yolo11n model, so that it can be deployed on terminals such as vehicle terminals and drones and has a good accuracy rate.
[0074] As an example, as Figure 3 shown Figure 3 is the overall structure diagram of the target detection model. The basic yolo11n network structure includes an input module, a Backbone module, a Neck module, and a detection module.
[0075] The Backbone module includes a conv convolutional layer and a C3k2 deformable convolutional network, which are used to extract the image features of the target image.
[0076] The Neck module includes a concat fusion operation, an upsample upsampling operation layer, as well as a C3k2 deformable convolutional network and a conv convolutional layer, which are used to perform different-scale processing on the image features of the target image, extract different-scale features of the image features of the target image, and perform multi-scale fusion on the different-scale features of the image features of the target image.
[0077] The Detect detection module is used to recognize the target image based on the fused feature map to obtain the accompanying target object. That is, the Detect module is the part in the yolo11n model for final target detection and classification. It predicts the category and locates the position of the accompanying target object in the image based on the previously extracted and fused feature map.
[0078] In some embodiments, when the drone is flying at a relatively high altitude during accompanying flight by visual detection, the accompanying target object is small and difficult to detect. Therefore, it is optimized by adding shallow features.
[0079] As an example, the target detection model further includes a shallow module.
[0080] It should be noted that the yolo11n network structure includes layers with different downsampling multiples, and layers with different downsampling multiples can capture features of different scales. These layers are arranged in a certain order in the Backbone module or other related modules of the model. Starting from the input layer, as the network depth increases, the downsampling multiple gradually increases. For example, Figure 3 as shown, taking the yolo11n network as an example, it includes the L4 layer with a downsampling of 4 times, the L8 layer with a downsampling of 8 times, the L16 layer with a downsampling of 16 times, and the L32 layer with a downsampling of 32 times. From the L4 layer to the L8 layer, then to the L16 layer and the L32 layer, each layer performs further feature extraction and downsampling operations based on the previous layer.
[0081] Since the accompanying target object with a small size occupies a relatively small proportion in the target image, a high-resolution feature map is required to capture its details, such as the edges of small targets (accompanying target objects with a size smaller than the threshold) and small texture features. However, the layer with a lower downsampling multiple (such as the L4 layer) has a relatively high resolution and can retain more detailed information, which is suitable for detecting small targets. Therefore, a shallow module connected to the shallow Backbone module is set in the yolo11n network structure.
[0082] The method further includes:
[0083] Performing feature extraction on the image features obtained after being processed by the Backbone module to obtain a shallow feature map;
[0084] Based on the shallow feature map, detecting the accompanying target object in the target image with a size smaller than the threshold.
[0085] The Backbone module is usually composed of multiple convolutional layers, pooling layers and other operations. As the number of network layers increases, the input target image will be processed, and features at different levels and different degrees of abstraction will be gradually extracted. In this process, relatively shallow features are output in the early stage to obtain shallow feature maps. The shallow feature maps retain more detailed information of the original image, such as local features like the edges, textures, and colors of objects in the image, but have relatively less semantic information and a higher resolution of the feature maps.
[0086] Therefore, small-sized accompanying target objects in the target image are detected through the shallow feature maps.
[0087] In this embodiment, by adding a shallow module to the target detection model, since the downsampling factor of this shallow module is relatively low and not much detail of small-resolution objects is lost, it is more likely to clearly capture the rich detail features of the accompanying target objects on the shallow feature maps, thereby improving the recognition accuracy of the target detection model for small targets.
[0088] As another example, the shallow module is connected after the deep feature repair module, and the shallow module is connected to the Neck module;
[0089] The method further includes:
[0090] The image features repaired by the feature repair module are input into the Neck module;
[0091] The shallow feature maps extracted by the shallow module are input into the Neck module;
[0092] The Neck module fuses the repaired image features and the shallow feature maps to detect the accompanying target objects in the target image.
[0093] The feature repair module is a structure in the improved target detection model, which can repair the occluded part for the occlusion problem of the accompanying target objects and improve the detection accuracy of the target detection model. The feature repair module will be described in detail in the subsequent chapters.
[0094] After receiving the repaired image features and the shallow feature maps, the Neck module will use various fusion methods to integrate them. The fusion methods can include but are not limited to operations such as concatenation (Concat). For example, through the concatenation operation, the shallow feature maps can be concatenated with the repaired image feature maps along the channel dimension, so that the new feature maps fuse the feature information of both aspects in the channel dimension, forming a comprehensive feature map containing multi-scale and multi-nature features.
[0095] The fused features combine the integrity of the features of the restored image (the overall features of the accompanying target are restored by the feature restoration module in the case of occlusion of the accompanying target) and the detail advantages of the shallow feature map (helping to capture the subtle features of the accompanying target, which is beneficial for detecting and accurately positioning small-sized accompanying targets). Based on such rich and optimized features, the Neck module will subsequently pass them to the Detect module. In the Detect module, through a series of operations such as convolution and classification, the fused features will be used to search for the regions in the target image that conform to the features of the accompanying target, determine its category, and locate its specific position in the image, so as to achieve accurate detection of the accompanying target in the target image.
[0096] In this embodiment, the shallow feature map and the feature data of the Backbone module are connected to achieve complementary information. By introducing the multi-scale feature fusion technology, features can be extracted from different levels, enhancing the representation ability of small targets and improving the recognition accuracy of the model for small targets.
[0097] In some other embodiments, since the vehicle is easily partially occluded by buildings or plants during accompanying flight through visual detection, an optimization is carried out by adding a feature restoration module.
[0098] As an example, a feature restoration module is added to the basic yolo11n network structure. The feature restoration module is connected between the Backbone module and the Neck module, as Figure 3 shown. The feature restoration module is used to repair the image features in the case of image feature loss, where the case of image feature loss includes partial occlusion of the accompanying target. That is, the improved object detection model can use the feature restoration module based on the generative adversarial network to repair the missing features for the occlusion problem of the accompanying target.
[0099] As an example, the feature restoration module includes a feature restoration network based on the generative adversarial network. The feature restoration network includes a generator, a local discriminator, and a global discriminator. The object detection model including the feature restoration module is trained. The process is as follows:
[0100] Generally, whether the network can learn good features has an important impact on the detection effect of the model. However, in actual detection, the occlusion of the identifier by plants or buildings causes noise to be mixed into the features of the identifier, affecting the final recognition result. Therefore, this case proposes a method of feature restoration based on the generative adversarial network for the feature loss caused by occlusion. The core idea is to continuously generate new values for the occlusion noise and adversarially make the identifier features containing occlusion noise gradually approach the real identifier features. As Figure 4As shown in the figure, the GAN-FRM network, a feature repair network structure based on the generative adversarial network, consists of a generator, a local discriminator, and a global discriminator. The role of the generator is to generate new feature values for the occluded parts. The local discriminator is used to compare the differences between the occluded parts and the real parts after feature repair, while the global discriminator compares the differences between the entire image after feature repair and the real image. The generator and the two discriminators iterate until the two discriminators can no longer determine whether the newly generated features are the real marker data features.
[0101] Among them, for the generator.
[0102] As Figure 4 shown, the generator mainly includes two parts: encoding and decoding of the image. First, the image inputs a vehicle with occlusion and markers (during the training process, incomplete data features are generated by randomly discarding pixel values from the complete picture, and the discarded pixel areas are random irregular shapes). The generator learns the image features through the encoding network and generates new features for it. The encoding network selects the Conv1 layer to the pool2 layer of the VGG16-Net. After encoding, two fully connected layers are added to transfer the encoded features to the decoding network. To ensure the accuracy of the image, the decoding network decodes the newly generated features into an image and sends it to the discriminator for comparison with the original image. The structure of the decoding network is symmetric to that of the encoding network, except that the pooling layer in the decoding network is replaced by an un-pooling layer.
[0103] For the local discriminator and the global discriminator.
[0104] The image generated by the generator is a restoration of the missing features, but the generated features may not effectively represent their corresponding original images. An excellent adversarial generation network can not only make the generated missing pictures as similar as possible to the real pictures, but also describe the relationship between the vehicle, the markers, and the surrounding environment. Therefore, in this case, a local feature discriminator and a global feature discriminator are designed. The local feature discriminator and the global feature discriminator use the same resnet-18 structure, and a sigmoid binary classifier is added at the end of the network to judge true or false. Among them, the local feature discriminator mainly judges the local details of the occluded area, while the global feature discriminator mainly makes the image after network repair as close as possible to the surrounding environment in terms of the global structure.
[0105] For the loss function.
[0106] In the feature repair model based on the generative adversarial network, in order to make the generated features as close as possible to the real features, this case uses the Euclidean distance method to calculate the difference between the generated features and the real features:
[0107]
[0108] Where M is the number of pixel points on the feature map, x i is the true feature value where the identifier has not been repaired by the generative adversarial network, x i ′ is the new feature value generated after the occluded part has been repaired by the generative adversarial network.
[0109] The local discriminator and the global discriminator have the same loss function, and the calculation formula is:
[0110]
[0111] Where L LD is the loss function of the local discriminator, and L GD is the loss function of the global discriminator, is the true data distribution, is the distribution of the occlusion noise, D(x) is the discrimination result of the data, Z is the noise (the part of the occluded or discarded pixels), and G(Z) is the data value generated by the noise.
[0112] The loss function L of this network is jointly composed of the generator and the discriminator, and its calculation formula is:
[0113] L = L G + L LD + L GD
[0114] Through the above process, the improved yolo11n detection model is trained, and the trained object detection model is used to identify the accompanying target objects in the target image. During the recognition process, when it is detected that the accompanying target object is locally occluded, the object detection model performs feature repair on the accompanying target object based on the feature repair method.
[0115] Specifically, when the object detection model detects that the accompanying target object is locally occluded, the feature repair module in the model (such as the GAN-FRM module based on the generative adversarial network) will be activated. The generator tries to generate reasonable features for the occluded area according to the features of the unoccluded part and the prior knowledge of the target object. The discriminator evaluates the generated features from local and global perspectives. Through multiple iterative trainings, the generator can generate more and more accurate features for the occluded part, so as to achieve a complete feature representation of the accompanying target object and improve the accuracy of object detection.
[0116] As an example, the feature repair method includes the following steps:
[0117] Step a1, generate new feature values for the occluded part of the accompanying target object to obtain the first repaired image.
[0118] In the target detection scenario, when the accompanying target object is occluded, in order to restore its complete recognizable features, it is necessary to generate new feature values for the occluded part. The generator in the trained GAN-FRM network generates new feature values that match the occluded part based on the feature information of the unoccluded part of the accompanying target object.
[0119] Fill the generated new feature values into the corresponding occluded positions of the accompanying target object, thereby constructing a repaired first image. The accompanying target object in the first image is more complete compared to the original target image with occlusion, making the feature representation of the entire target object in the image closer to the true state when it is not occluded.
[0120] Step a2, perform local discrimination on the first image to obtain a first output result.
[0121] Local discrimination mainly focuses on the difference between the occluded part and the real part of the accompanying target object in the repaired first image to judge the quality of the occlusion repair effect. Then, after analyzing and judging the above local features, the local discriminator will output a result, that is, the first output result.
[0122] Step a3, perform global discrimination on the target image where the first image is located after repair to obtain a second output result.
[0123] Global discrimination is from a more macroscopic perspective, comparing the differences between the entire image after feature repair and the real image, focusing on considering the rationality of the features by placing the repaired first image in the overall environment of the entire target image, that is, paying attention to the coordination between the accompanying target object and the surrounding environment after repair. After evaluating the entire target image, the global discriminator will output a second output result.
[0124] Step a4, obtain the image features of the repaired image according to the first output result and the second output result.
[0125] After obtaining the first output result and the second output result, a comprehensive judgment will be made on the previously generated repaired first image based on these two results. If both results indicate that the repaired features are reasonable at both the local and global levels (for example, the discrimination result is true), then it can be determined that the current repair effect is good, and directly use the features corresponding to the repaired first image as the final image features of the repair.
[0126] However, if there is anything unreasonable in the output result of any of the two discriminators (for example, any output result is false), these feedback messages will be passed back to the generator, prompting the generator to adjust its strategies and parameters for generating new feature values, regenerate the feature values for the occluded part, and perform operations such as local discrimination and global discrimination again. Through multiple iterations, the repair effect is continuously optimized until the repaired image features that meet the requirements are obtained from both local and global perspectives, so as to ensure that the features of the accompanying target object can be accurately and completely restored, thereby improving the accuracy and reliability of subsequent target detection.
[0127] Step 300: According to the position information of the accompanying target object in the target image after repair, determine the azimuth information of the accompanying target object relative to the UAV, and control the UAV to continuously accompany the accompanying target object.
[0128] The target detection model identifies the complete accompanying target object, and represents the position information of the accompanying target object in the target image through the coordinate information of the complete outer contour of the accompanying target object in the image coordinate system. According to the position information in the image and the attitude information of the UAV itself and / or the installation attitude information of the camera on the UAV, calculate the azimuth information of the accompanying target object relative to the UAV.
[0129] Based on the calculated azimuth information, the flight control system of the UAV will generate corresponding control commands to adjust the flight attitude and speed of the UAV to keep it continuously accompanying the accompanying target object.
[0130] Furthermore, during the accompanying process, the UAV can also be controlled according to the distance between the accompanying target object and the UAV.
[0131] As an example, for tasks with low requirements for distance accuracy, the distance between the UAV and the accompanying target object can be estimated according to the size of the accompanying target object in the target image. As another example, for tasks with high requirements for distance accuracy, add a distance detection model or use binocular ranging to determine the distance between the accompanying target object and the UAV.
[0132] By maintaining the relative position stability with the target object through the above azimuth information and distance information, it is ensured that when the target object moves or the UAV itself is affected by external interference, it can still accurately follow the accompanying target object and achieve a reliable accompanying task.
[0133] In some other embodiments, the method further includes:
[0134] When the accompanying target object in the target image is completely occluded, obtain the GPS signal of the accompanying target object; determine the azimuth information of the accompanying target object relative to the UAV according to the GPS signal, and control the UAV to continuously accompany the accompanying target object.
[0135] When the accompanying target object is completely blocked in the target image captured by the UAV, it means that it is no longer possible to directly determine its position and state relying on traditional target detection methods based on image visual features. In order to continue to achieve the tracking and accompanying of the accompanying target object, the GPS signal of the GPS device carried by the accompanying target object itself can provide high-precision positioning information.
[0136] In this embodiment, an accompanying strategy that mainly uses the vision scheme and supplements with GPS signals is adopted. When the vision signal is interrupted for a long time, GPS is used for supplementation. This not only effectively overcomes the disadvantages of GPS signals being vulnerable to interference and inaccurate positioning in urban environments, but also gives full play to the advantages of vision technology. Through the complementarity of the two, the stability and reliability of the accompanying task are ensured, greatly improving the accuracy and adaptability of the accompanying operation in complex urban environments, reducing accompanying mistakes caused by signal problems, and ensuring the smooth execution of the task.
[0137] This application provides a vehicle-mounted UAV accompanying control method, device, and computer-readable storage medium. Compared with the current implementation of UAV accompanying through satellite positioning technology, where GPS signals are vulnerable to interference and inaccurate in the city, the target image captured by the camera on the UAV is obtained; the target image is recognized through a target detection model to determine the accompanying target object. By means of visual recognition, it is possible to achieve precise accompanying of the UAV in different scenarios without relying on external network data, solving the problems that occur when accompanying based on GPS signals currently. Moreover, when it is detected that the accompanying target object is partially blocked, the target detection model is used to perform feature repair on the accompanying target object to improve the detection accuracy of the detection network for the accompanying target object. When the target detection model detects the repaired accompanying target object, according to the position information of the repaired accompanying target object in the target image, the azimuth information of the accompanying target object relative to the UAV is determined, and the UAV is controlled to continuously accompany the accompanying target object.
[0138] Based on the same application concept as the above method, the embodiment of this application also proposes a vehicle-mounted UAV accompanying control device, as Figure 5 shown.
[0139] The device includes:
[0140] An image acquisition module 502, configured to acquire a target image captured by a camera on the UAV;
[0141] An image recognition module 504, configured to recognize the target image through a target detection model to determine the accompanying target object; wherein, when it is detected that the accompanying target object is partially blocked, the target detection model is used to perform feature repair on the accompanying target object;
[0142] The accompanying flight control module 506 is configured to determine the orientation information of the accompanying flight target relative to the drone according to the position information of the accompanying flight target in the target image after repair, and control the drone to perform continuous accompanying flight on the accompanying flight target.
[0143] For the specific implementation process of the functions and roles of each module / sub-module / unit in the above device, please refer to the implementation process of the corresponding steps in the above method, which can achieve the same technical effect and will not be elaborated here.
[0144] Figure 6 An example of the physical structure diagram of a vehicle-mounted drone accompanying flight control device is shown as Figure 6 As shown, the vehicle-mounted drone accompanying flight control device may include: a processor 810, a communication interface 820, a memory 830, and a communication bus 840. Among them, the processor 810, the communication interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 can call the logical instructions in the memory 830 to execute the vehicle-mounted drone accompanying flight control method.
[0145] In addition, when the logical instructions in the above-mentioned memory 830 are implemented in the form of software functional units and sold or used as an independent product, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present application. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.
[0146] On the other hand, the present application also provides a computer program product. The computer program product includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the vehicle-mounted drone accompanying flight control method provided by the above-mentioned methods.
[0147] On yet another aspect, the present application also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it is implemented to execute the vehicle-mounted drone accompanying flight control method provided by the above-mentioned methods.
[0148] It should be noted that the technical solutions or technical features described in the above embodiments can be combined or supplemented with each other without conflict. The scope of protection of this application is not limited to the precise structures described in the above embodiments and shown in the drawings; any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of this application shall be included within the scope of protection of this application.
Claims
1. A vehicle-mounted UAV accompanying flight control method, characterized in that: The method comprises: Get the target image taken by the camera on the drone; The target image is identified by a target detection model to determine the accompanying target; wherein, when it is detected that the accompanying target is partially blocked, the target detection model is used to perform feature repair on the accompanying target; According to the restored position information of the accompanying target in the target image, the orientation information of the accompanying target relative to the unmanned aerial vehicle is determined, and the unmanned aerial vehicle is controlled to continuously accompany the accompanying target.
2. The vehicle-mounted UAV accompanying flight control method according to claim 1, characterized in that: The target detection model includes a Backbone module, a feature repair module, a Neck module, and a detection module; the feature repair module is connected between the Backbone module and the Neck module; The Backbone module is used to extract image features of the target image; The feature repair module is used to repair the image feature in the case of image feature loss, wherein the image feature loss includes that the accompanying target is partially blocked; The Neck module is used to perform different scale processing on the restored image features, extract the features of different scales of the restored image features, and perform multi-scale fusion on the features of different scales of the restored image features; The detection module is used to identify the target image based on the fused feature map to obtain a restored and complete accompanying flight target.
3. The vehicle-mounted UAV accompanying flight control method according to claim 2, characterized in that: The feature restoration module is based on a feature restoration network and according to a feature restoration method to restore the image features in the case of image feature loss; Wherein, the feature repair method comprises: Generate a new feature value for the blocked portion of the accompanying target object to obtain a restored first image; Performing local discrimination on the first image to obtain a first output result; Performing global discrimination on the restored target image where the first image is located to obtain a second output result; The restored image features are obtained according to the first output result and the second output result.
4. The vehicle-mounted UAV accompanying flight control method according to claim 3, characterized in that: The feature restoration module includes a feature restoration network based on a generative adversarial network, and the feature restoration network includes a generator, a local discriminator, and a global discriminator; Generate new feature values for the occluded part by the generator; Comparing the difference between the occluded part and the real part of the new feature value by the local discriminator; Compare the differences between the whole image after feature restoration and the real markers in the real image through the global discriminator; The generator and the two discriminators are iterated until the two discriminators cannot determine whether the generated new feature value is the feature of the real marker, thereby obtaining the restored image feature.
5. The vehicle-mounted UAV accompanying flight control method according to claim 2, characterized in that: The target detection model also includes a shallow module; the shallow module is connected to the shallow Backbone module; The method further comprises: Extracting features of the image obtained after being processed by the Backbone module to obtain a shallow feature map; Based on the shallow feature map, the accompanying flying target whose size is smaller than a threshold value in the target image is detected.
6. The vehicle-mounted UAV accompanying flight control method according to claim 5, characterized in that: After the shallow module is connected to the deep feature repair module, the shallow module is connected to the Neck module; The method further comprises: The image features restored by the feature restoration module are input into the Neck module; The shallow feature map extracted by the shallow module is input into the Neck module; The repaired image features are fused with the shallow feature map through the Neck module to detect the accompanying flying target in the target image.
7. The vehicle-mounted UAV accompanying flight control method according to any one of claims 1 to 6, characterized in that: The method further comprises: The accompanying flying target has a plurality of nested icons, wherein the sizes of the plurality of icons are set in a step-by-step manner according to the recognition accuracy of the target detection model.
8. The vehicle-mounted UAV accompanying flight control method according to claim 1, characterized in that: The method further comprises: The target detection model includes a yoloN network model; wherein the yoloN network model includes a yolo11n model.
9. The vehicle-mounted UAV accompanying flight control method according to claim 1, characterized in that: The method further comprises: When the accompanying flying target is completely blocked in the target image, obtaining a GPS signal of the accompanying flying target; The position information of the accompanying target object relative to the unmanned aerial vehicle is determined according to the GPS signal, and the unmanned aerial vehicle is controlled to continuously accompany the accompanying target object.
10. A vehicle-mounted UAV companion flight control device, characterized in that: It includes a memory, a processor, and a vehicle-mounted UAV companion flight control program stored in the memory and executable on the processor. When the processor executes the vehicle-mounted UAV companion flight control program, the steps of the vehicle-mounted UAV companion flight control method as described in any one of claims 1 to 9 are implemented.
11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a vehicle-mounted UAV companion flight control program, and when the vehicle-mounted UAV companion flight control program is executed, the steps of the vehicle-mounted UAV companion flight control method as described in any one of claims 1-9 are implemented.