Unmanned aerial vehicle control method and apparatus based on unmanned aerial vehicle vision, device and medium

Abstract

Embodiments of the present disclosure disclose a method, device, equipment and medium for unmanned aerial vehicle (UAV) control based on UAV vision. A specific embodiment of the method comprises: a control signal emitting device emits signals in real time; a control recognition device performs recognition processing on the unknown flying object to generate a recognition result; a monitoring UAV in a UAV group performs image tracking processing on the unknown UAV; the unknown UAV is positioned; a second unknown UAV trajectory is generated, and the first unknown UAV trajectory is corrected according to the second unknown UAV trajectory to obtain a corrected UAV trajectory; the unknown UAV is subjected to trajectory prediction processing to generate a UAV predicted trajectory; and the UAV group or the interception device intercepts the unknown UAV. This embodiment improves the regional security of the detection area.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure relate to the field of computer technology, and in particular, to a method and apparatus for controlling a UAV based on UAV vision, an electronic device, and a computer readable medium. BACKGROUND

[0002] With the continuous development of UAV technology, how to control unknown UAVs has become an important research topic. At present, when controlling unknown UAVs, the commonly used method is to control the unknown UAVs by electromagnetic interference.

[0003] However, when the above method is used to control unknown UAVs, the following technical problems often exist:

[0004] Unknown UAVs may be illegally modified (for example, adding a signal anti-interference module), and the electromagnetic interference method cannot effectively control the UAVs, resulting in low regional safety of the detection area.

[0005] The above information disclosed in this BACKGROUND section is only for the purpose of enhancing the understanding of the background of the present inventive concepts, and therefore, it can contain information that does not form the prior art that is known to those of ordinary skill in the art in the country. SUMMARY

[0006] The summary section of the present disclosure is used to introduce the concepts in a brief manner, which will be described in detail in the specific embodiments section. The summary section of the present disclosure is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to be used to limit the scope of the claimed technical solutions.

[0007] Some embodiments of the present disclosure propose a method and apparatus for controlling a UAV based on UAV vision, an electronic device, and a computer readable medium, to solve one or more of the technical problems mentioned in the background section.

[0008] In a first aspect, some embodiments of the present disclosure provide a method for controlling unmanned aerial vehicles (UAVs) based on UAV vision, the method comprising: controlling a signal emitting device to emit signals in real time to monitor unregistered UAVs; in response to receiving a reflected signal representing that an unknown flying object is monitored, controlling an identification device to perform identification processing on the unknown flying object to generate an identification result; in response to the identification result representing that the unknown flying object is an unknown UAV, controlling a monitoring UAV in a UAV group to perform image tracking processing on the unknown UAV to generate an unknown UAV image sequence; performing positioning processing on the unknown UAV according to the reflected signal received in real time to generate a first unknown UAV trajectory; generating a second unknown UAV trajectory according to the positioning of the monitoring UAV and the unknown UAV image sequence, and performing correction processing on the first unknown UAV trajectory according to the second unknown UAV trajectory to obtain a corrected UAV trajectory; performing trajectory prediction processing on the unknown UAV according to the corrected UAV trajectory to generate a UAV predicted trajectory; and controlling the UAV group or an interception device to perform interception processing on the unknown UAV according to the UAV predicted trajectory.

[0009] In a second aspect, some embodiments of the present disclosure provide a device for controlling UAVs based on UAV vision, the device comprising: a first control unit configured to control a signal emitting device to emit signals in real time to monitor unregistered UAVs; a second control unit configured to, in response to receiving a reflected signal representing that an unknown flying object is monitored, control an identification device to perform identification processing on the unknown flying object to generate an identification result; a third control unit configured to, in response to the identification result representing that the unknown flying object is an unknown UAV, control a monitoring UAV in a UAV group to perform image tracking processing on the unknown UAV to generate an unknown UAV image sequence; a positioning unit configured to perform positioning processing on the unknown UAV according to the reflected signal received in real time to generate a first unknown UAV trajectory; a generating unit configured to generate a second unknown UAV trajectory according to the positioning of the monitoring UAV and the unknown UAV image sequence, and perform correction processing on the first unknown UAV trajectory according to the second unknown UAV trajectory to obtain a corrected UAV trajectory; a trajectory prediction unit configured to perform trajectory prediction processing on the unknown UAV according to the corrected UAV trajectory to generate a UAV predicted trajectory; and a fourth control unit configured to control the UAV group or an interception device to perform interception processing on the unknown UAV according to the UAV predicted trajectory.

[0010] In a third aspect, some embodiments of the present disclosure provide an electronic device, comprising: one or more processors; a storage device having one or more programs stored thereon, when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner of the first aspect.

[0011] In a fourth aspect, some embodiments of the present disclosure provide a computer readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method described in any implementation manner of the first aspect.

[0012] The above various embodiments of the present disclosure have the following beneficial effects: through the unmanned aerial vehicle control method based on unmanned aerial vehicle vision of some embodiments of the present disclosure, the area safety of the detection area is improved. Specifically, the reason why the area safety of the detection area is low is that unknown unmanned aerial vehicles may be illegally modified (for example, signal anti-interference modules are added), and electromagnetic interference cannot be used for effective unmanned aerial vehicle control, resulting in low area safety of the detection area. Based on this, the unmanned aerial vehicle control method based on unmanned aerial vehicle vision of some embodiments of the present disclosure first controls the signal emitting device to emit signals in real time to monitor unknown unmanned aerial vehicles. In this way, it can be detected in real time whether there are unknown unmanned aerial vehicles in the detection area. Second, in response to receiving a reflection signal representing that an unknown flying object is monitored, the identification device is controlled to identify the unknown flying object to generate an identification result. Then, in response to the identification result representing that the unknown flying object is an unknown unmanned aerial vehicle, the monitoring unmanned aerial vehicle in the unmanned aerial vehicle group is controlled to perform image tracking processing on the unknown unmanned aerial vehicle to generate an unknown unmanned aerial vehicle image sequence. In this way, the unknown unmanned aerial vehicle can be tracked and photographed by the associated unmanned aerial vehicle, and clearer images of the unknown unmanned aerial vehicle can be obtained. Then, according to the reflection signal received in real time, the unknown unmanned aerial vehicle is positioned to generate a first unknown unmanned aerial vehicle trajectory; according to the positioning of the monitoring unmanned aerial vehicle and the unknown unmanned aerial vehicle image sequence, a second unknown unmanned aerial vehicle trajectory is generated, and the first unknown unmanned aerial vehicle trajectory is corrected according to the second unknown unmanned aerial vehicle trajectory to obtain a corrected unmanned aerial vehicle trajectory. In this way, the corrected unmanned aerial vehicle flight trajectory can be determined. Then, according to the corrected unmanned aerial vehicle trajectory, the unknown unmanned aerial vehicle is trajectory predicted to generate an unmanned aerial vehicle predicted trajectory. In this way, the unknown unmanned aerial vehicle can be trajectory predicted through the flight trajectory of the unknown unmanned aerial vehicle. Finally, according to the unmanned aerial vehicle predicted trajectory, the unmanned aerial vehicle group or the interception device is controlled to intercept the unknown unmanned aerial vehicle. In this way, the unknown unmanned aerial vehicle can be intercepted, and thus the area safety of the detection area can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0013] The above and other features, advantages, and aspects of embodiments of the present disclosure will become more apparent by describing in detail some embodiments thereof with reference to the annexed drawings in which: like reference numerals refer to like elements throughout. The annexed drawings are schematic and are not intended to accurately depict the proportions or relative positions of elements.

[0014] Figure 1 is a flowchart of some embodiments of a UAV control method based on UAV vision according to the present disclosure;

[0015] Figure 2 is a structural schematic diagram of some embodiments of a UAV control device based on UAV vision according to the present disclosure;

[0016] Figure 3 is a structural schematic diagram of an electronic device suitable for use to implement some embodiments of the present disclosure;

[0017] Figure 4 is a schematic diagram of a UAV interception system according to some embodiments of a UAV control method based on UAV vision according to the present disclosure;

[0018] Figure 5 is a trajectory display test diagram according to some embodiments of a UAV control method based on UAV vision according to the present disclosure. DETAILED DESCRIPTION

[0019] Embodiments of the present disclosure will be described in more detail with reference to the drawings. Although some embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure will be more thoroughly and completely understood. It should be understood that the drawings and embodiments of the present disclosure are only for illustrative purposes and are not intended to limit the scope of protection of the present disclosure.

[0020] It should also be noted that, for the sake of brevity, only some of the pertinent features of the application are shown in each drawing. Embodiments of the present disclosure and features in embodiments can be combined with each other as long as there is no conflict.

[0021] It should be noted that the terms "first", "second", and the like in the present disclosure are used only to distinguish different devices, modules, or units, and do not imply the order or interdependence of the functions performed by these devices, modules, or units.

[0022] It should be noted that the terms "one", "multiple" in the present disclosure are illustrative and not restrictive, and those skilled in the art should understand that, unless otherwise explicitly stated in the context, it should be understood as "one or more".

[0023] Names of messages or information exchanged between multiple devices in the embodiments of the present disclosure are only for illustrative purposes, and are not intended to limit the scope of the messages or information.

[0024] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0025] Figure 1 Flow 100 of some embodiments of a UAV control method based on UAV vision according to the present disclosure is shown. The UAV control method based on UAV vision includes the following steps:

[0026] Step 101, control the signal emitting device to emit signals in real time to monitor unregistered UAVs.

[0027] In some embodiments, the execution subject (e.g., server) of the UAV control method based on UAV vision can control the signal emitting device to emit signals in real time to monitor unregistered UAVs. Wherein the UAV control method based on UAV vision can be applied to a UAV control system. The UAV control system can include a signal emitting device, a UAV group, an identification device, and an interception device. The signal emitting device can be a detection radar. The UAV group can be a UAV group connected to the UAV control system wirelessly. The identification device can be a terminal with a shooting function for identifying the photographed object. The interception device can be used to intercept objects in the air. Figure 4 A schematic diagram of a UAV interception system deployed in a server is shown. As Figure 4 shown, this system platform adopts a C / S structure design, can load GIS high-definition maps, and has functions such as multi-target intrusion alarm, target accurate positioning, display of target motion trajectory and flight control hand position.

[0028] Step 102, in response to receiving a reflection signal representing monitoring an unknown flying object, control the identification device to identify the unknown flying object to generate an identification result.

[0029] In some embodiments, the execution subject can control the identification device to identify the unknown flying object to generate an identification result in response to receiving a reflection signal representing monitoring an unknown flying object.

[0030] In practice, the identification device can be controlled to identify the unknown flying object by the following steps:

[0031] First, control the identification device to shoot the unknown flying object to generate a shooting image sequence. Wherein the shooting process can be shooting the unknown flying object every interval of a preset time length.

[0032] In the second step, the identification model inputs each of the photographed image sequence into a pre-trained drone identification model to obtain an identification result. The drone identification model can be a pre-trained model for identifying drones. For example, the drone identification model can be a classification model trained using a preset data set including drone images.

[0033] In the process of solving the above technical problems by adopting the technical solutions, the following problems are often accompanied: when identifying whether the flying object is a drone, the flying object is small in size and there is a certain distance, so the flying object cannot be accurately identified, leading to the unknown drone being identified as other objects, and further leading to low regional safety of the detection area.

[0034] In some optional implementations of some embodiments, the execution subject can identify the unknown flying object by the following steps:

[0035] In the first step, based on the photographed image sequence, the following processing steps are performed:

[0036] In the first processing step, the first photographed image in the photographed image sequence is subjected to motion target identification processing to generate a photographed foreground target image. The motion target identification processing can be to determine the moving foreground target by the ViBe algorithm.

[0037] In the second processing step, the moving target displayed in the photographed foreground target image is subjected to contour detection processing to generate a detected target image. Here, the contour detection algorithm based on OpenCV can be used to detect the contour of the moving target.

[0038] In the third processing step, the position of the moving target in the first photographed image is determined according to the detected target image to obtain image position information.

[0039] In the fourth processing step, the first photographed image is segmented according to the image position information to generate a segmented photographed image. In practice, the position represented by the image position information can be taken as the center, and a preset length can be taken as the radius to segment the first photographed image to generate a segmented photographed image.

[0040] In the fifth processing step, the segmented photographed image is subjected to feature extraction processing to generate a moving target feature vector.

[0041] In practice, the segmented photographed image can be subjected to feature extraction processing by the following sub-steps to generate a moving target feature vector:

[0042] A first sub-step, performing gamma correction on the segmented captured image to generate a corrected captured image. Here, the image size of the segmented captured image needs to be adjusted before gamma correction.

[0043] A second sub-step, determining gradient information of each pixel in the corrected captured image to obtain a gradient information set. The gradient information is used to represent the contour shape of the local image.

[0044] A third sub-step, combining each pixel in the corrected captured image to generate at least one pixel unit.

[0045] A fourth sub-step, determining a feature descriptor of each pixel unit in the at least one pixel unit according to the gradient information set.

[0046] A fifth sub-step, combining each preset number of pixel units in the at least one pixel unit to generate at least one pixel block, and concatenating the feature descriptors corresponding to each pixel unit included in each pixel block to generate a combined feature descriptor.

[0047] A sixth sub-step, concatenating each generated combined feature descriptor into a motion target feature vector.

[0048] A sixth processing step, inputting the motion target feature vector into a pre-trained motion target recognition model to obtain a recognition result. The motion target recognition model can be a pre-trained nonlinear SVM model.

[0049] Second, in response to not recognizing the presence of a motion target in the first captured image, and the captured image sequence after deleting the first captured image is not empty, taking the captured image sequence after deleting the first captured image as the captured image sequence, and executing the above processing steps again.

[0050] The first step-second step as an invention point of an embodiment of the present disclosure, combined with the following step "step 107", solves the technical problem "when identifying whether the flying object is a drone, due to the small size of the flying object and the existence of a certain distance, the flying object cannot be accurately identified, leading to the identification of unknown drones as other objects, and further leading to the low regional safety of the detection area". The reason for the low regional safety of the detection area is as follows: when identifying whether the flying object is a drone, due to the small size of the flying object and the existence of a certain distance, the flying object cannot be accurately identified, leading to the identification of unknown drones as other objects, and further leading to the low regional safety of the detection area. If the above factors are solved, the effect of improving the regional safety of the detection area can be achieved. In order to achieve this effect, the present disclosure first, based on the photographed image sequence, performs the following processing steps: performing motion target identification processing on the first photographed image in the photographed image sequence to generate a photographed foreground target image. Thus, it can be determined whether the unknown flying object is in motion. Second, the motion target displayed in the photographed foreground target image is subjected to contour detection processing to generate a detected target image. Thus, the contour of the motion target can be determined. Third, according to the detected target image, the position of the motion target in the first photographed image is determined to obtain image position information. Thus, the position of the motion target in the photographed image can be determined. Fourth, according to the image position information, the first photographed image is subjected to segmentation processing to generate a segmented photographed image; the segmented photographed image is subjected to feature extraction processing to generate a motion target feature vector. Thus, the feature information of the segmented image can be extracted, the identification range is reduced to the sub-image area, and the identification speed and accuracy are improved. Further, it can avoid identifying unknown drones as other objects, and improve the regional safety of the detection area. Fifth, the segmented photographed image is subjected to gamma correction processing to generate a corrected photographed image; the gradient information of each pixel point in the corrected photographed image is determined to obtain a gradient information set. Thus, the feature information of the local image can be determined. Sixth, each pixel point in the corrected photographed image is subjected to combination processing to generate at least one pixel unit; according to the gradient information set, the feature descriptor of each pixel unit in at least one pixel unit is determined; each preset number of pixel units in at least one pixel unit is subjected to combination processing to generate at least one pixel block, and the feature descriptors corresponding to each pixel unit included in each pixel block are concatenated to generate a combined feature descriptor; and each combined feature descriptor generated is concatenated into a motion target feature vector. Thus, the feature vector of the unknown flying object can be generated.Seventh, input the moving target feature vector into the pre-trained moving target recognition model to obtain a recognition result; in response to the fact that no moving target is recognized in the first captured image and the sequence of captured images after the first captured image is deleted is not empty, the sequence of captured images after the first captured image is deleted is taken as the sequence of captured images, and the above processing steps are executed again. In this way, the identification processing of the positional flying object is completed, the identification range of the unknown flying object is reduced, the identification speed and accuracy are improved, and the unknown unmanned aerial vehicle can be avoided from being identified as other objects, thereby improving the regional safety of the detection area.

[0051] Step 103, in response to the fact that the above-mentioned identification result indicates that the above-mentioned unknown flying object is an unknown unmanned aerial vehicle, the monitoring unmanned aerial vehicle in the above-mentioned unmanned aerial vehicle group is controlled to perform image tracking processing on the above-mentioned unknown unmanned aerial vehicle to generate an unknown unmanned aerial vehicle image sequence.

[0052] In some embodiments, the above-mentioned execution subject can control the monitoring unmanned aerial vehicle in the above-mentioned unmanned aerial vehicle group to perform image tracking processing on the above-mentioned unknown unmanned aerial vehicle to generate an unknown unmanned aerial vehicle image sequence in response to the fact that the above-mentioned identification result indicates that the above-mentioned unknown flying object is an unknown unmanned aerial vehicle. The image tracking processing can be to control the monitoring unmanned aerial vehicle to install a shooting device to perform real-time tracking shooting on the above-mentioned unknown unmanned aerial vehicle.

[0053] Step 104, performing positioning processing on the above-mentioned unknown unmanned aerial vehicle according to the real-time received reflection signal to generate a first unknown unmanned aerial vehicle trajectory.

[0054] In some embodiments, the above-mentioned execution subject can perform positioning processing on the above-mentioned unknown unmanned aerial vehicle according to the real-time received reflection signal to generate a first unknown unmanned aerial vehicle trajectory. The reflection signal can be a signal reflected back by the detection radar after detecting the unknown unmanned aerial vehicle. Figure 5 As shown in the trajectory display test diagram, the generated first unknown unmanned aerial vehicle trajectory can be displayed in the system to facilitate observation of the unmanned aerial vehicle trajectory.

[0055] In practice, the above-mentioned execution subject can perform positioning processing on the above-mentioned unknown unmanned aerial vehicle to generate a first unknown unmanned aerial vehicle trajectory by the following steps:

[0056] First, a preset real scene three-dimensional model is obtained. The preset real scene three-dimensional model is a three-dimensional model of a signal emission range corresponding to the signal emission device. The signal emission range can be a range of signals that can be emitted by the signal emission device.

[0057] Secondly, at least one reflection signal is selected from each reflection signal received in real time according to a preset time interval to obtain a reflection signal sequence. The preset time interval can be a preset time interval for selecting every two reflection signals. For example, the preset time interval can be 1 s.

[0058] Thirdly, for each reflection signal in the reflection signal sequence, a position represented by the reflection signal is marked in the preset real scene three-dimensional model. Here, for each reflection signal in the reflection signal sequence, coordinates and height represented by the reflection signal can be marked in the preset real scene three-dimensional model.

[0059] Fourthly, a first unknown unmanned aerial vehicle trajectory is generated according to the positions marked in the preset real scene three-dimensional model.

[0060] Step 105, a second unknown unmanned aerial vehicle trajectory is generated according to the positioning of the monitoring unmanned aerial vehicle and the unknown unmanned aerial vehicle image sequence, and the first unknown unmanned aerial vehicle trajectory is corrected according to the second unknown unmanned aerial vehicle trajectory to obtain a corrected unmanned aerial vehicle trajectory.

[0061] In some embodiments, the execution subject can generate a second unknown unmanned aerial vehicle trajectory according to the positioning of the monitoring unmanned aerial vehicle and the unknown unmanned aerial vehicle image sequence, and correct the first unknown unmanned aerial vehicle trajectory according to the second unknown unmanned aerial vehicle trajectory to obtain a corrected unmanned aerial vehicle trajectory.

[0062] Step 106, the unknown unmanned aerial vehicle is trajectory predicted according to the corrected unmanned aerial vehicle trajectory to generate an unmanned aerial vehicle predicted trajectory.

[0063] In some embodiments, the execution subject can generate an unmanned aerial vehicle predicted trajectory by trajectory predicting the unknown unmanned aerial vehicle according to the corrected unmanned aerial vehicle trajectory.

[0064] In practice, the execution subject can perform trajectory prediction on the unknown unmanned aerial vehicle by the following steps:

[0065] Firstly, a set of unmanned aerial vehicle trajectory points is determined according to the corrected unmanned aerial vehicle trajectory. In practice, a preset number of unmanned aerial vehicle trajectory points can be selected from the corrected unmanned aerial vehicle trajectory, and the preset number of unmanned aerial vehicle trajectory points are combined to form a set of unmanned aerial vehicle trajectory points. Here, the execution subject can select unmanned aerial vehicle trajectory points from the corrected unmanned aerial vehicle trajectory every preset time step to obtain a set of unmanned aerial vehicle trajectory points. The preset time step can be a preset time step of the unknown unmanned aerial vehicle flight.

[0066] Secondly, for each of the UAV trajectory points in the set of UAV trajectory points, a UAV speed corresponding to the UAV trajectory point and a UAV displacement corresponding to the UAV trajectory point are determined. The UAV speed can be the speed of the unknown UAV at the UAV trajectory point. The UAV displacement can be the displacement of the unknown UAV from a previous UAV trajectory point to the current UAV trajectory point.

[0067] Thirdly, according to the determined UAV speeds and UAV displacements, a UAV speed sequence and a UAV displacement sequence are generated. In practice, the execution subject can sort the UAV speeds and the UAV displacements in chronological order according to the time of the UAV trajectory points corresponding to the UAV speeds and the UAV displacements, and generate the UAV speed sequence and the UAV displacement sequence.

[0068] Fourthly, the UAV speed sequence and the UAV displacement sequence are normalized to generate a normalized UAV speed sequence and a normalized UAV displacement sequence.

[0069] Fifthly, the normalized UAV speed sequence and the normalized UAV displacement sequence are input into a pre-trained UAV trajectory prediction model to obtain a UAV predicted trajectory.

[0070] Optionally, the UAV trajectory prediction model can be trained by the following steps:

[0071] Firstly, a sample set is obtained.

[0072] In some embodiments, the execution subject can obtain a sample set. The samples in the sample set include sample normalized UAV speed sequences and sample normalized UAV displacement sequences, and sample UAV predicted trajectories corresponding to the sample normalized UAV speed sequences and the sample normalized UAV displacement sequences.

[0073] Secondly, a sample is selected from the sample set.

[0074] In some embodiments, the execution subject can select a sample from the sample set. Here, the execution subject can randomly select a sample from the sample set.

[0075] Thirdly, the sample is input into an initial network model to obtain a UAV predicted trajectory corresponding to the sample.

[0076] In some embodiments, the execution subject can input the sample into an initial network model to obtain a UAV predicted trajectory corresponding to the sample. The initial neural network can be a generative model capable of obtaining a UAV predicted trajectory according to a normalized UAV speed sequence and a normalized UAV displacement sequence.

[0077] A fourth step is to determine a loss value between the UAV predicted trajectory and a sample UAV predicted trajectory included in the sample.

[0078] In some embodiments, the execution subject can determine the loss value between the UAV predicted trajectory and the sample UAV predicted trajectory included in the sample. In practice, the loss value between the UAV predicted trajectory and the sample UAV predicted trajectory included in the sample can be determined based on a preset loss function. For example, the preset loss function can be a cross-entropy loss function.

[0079] A fifth step is to adjust the network parameters of the initial network model in response to the loss value being greater than or equal to a preset threshold.

[0080] In some embodiments, the execution subject can adjust the network parameters of the initial network model in response to the loss value being greater than or equal to a preset threshold. Here, the preset threshold is not limited. For example, a loss difference value can be obtained by subtracting the preset threshold from the loss value. On this basis, the error value is propagated from the last layer of the model to the front layer using methods such as back propagation and stochastic gradient descent to adjust the parameters of each layer. Of course, according to needs, the network freezing (dropout) method can also be used to keep some layers of network parameters unchanged and not adjusted, and no limitation is made on this.

[0081] Optionally, the initial network model is determined as a UAV trajectory prediction model in response to the loss value being less than the preset threshold.

[0082] In some embodiments, the execution subject can determine the initial network model as a UAV trajectory prediction model in response to the loss value being less than the preset threshold.

[0083] Step 107 is to control the UAV group or the interception device to intercept the unknown UAV according to the UAV predicted trajectory.

[0084] In some embodiments, the execution subject can control the UAV group or the interception device to intercept the unknown UAV according to the UAV predicted trajectory.

[0085] In practice, the execution subject can intercept the unknown UAV by the following steps:

[0086] In a first step, the size of the unknown UAV is predicted based on the unknown UAV image sequence to obtain a size prediction result. In practice, each unknown UAV image in the unknown UAV image sequence can be input into a pre-trained size prediction model to obtain a prediction result. Then, the mean of each prediction result is determined as the size prediction result.

[0087] In a second step, it is determined whether the UAV predicted trajectory meets a preset trajectory condition. The preset trajectory condition can be that the UAV predicted trajectory represents the unknown UAV flying into an interception range corresponding to the interception device.

[0088] In a third step, in response to the UAV predicted trajectory meeting the preset trajectory condition, the interception device included in the UAV control system is controlled to perform a catapult interception operation on the unknown UAV based on the UAV predicted trajectory. The catapult interception operation can be that the interception device launches an interception net towards the unknown UAV.

[0089] In a fourth step, in response to the UAV predicted trajectory not meeting the preset trajectory condition, the unknown UAV is dynamically intercepted by the UAV group based on the size prediction result and the UAV predicted trajectory. Here, in response to the size prediction result being greater than a preset UAV interception size,

[0090] The above various embodiments of the present disclosure have the following beneficial effects: through the unmanned aerial vehicle control method based on unmanned aerial vehicle vision of some embodiments of the present disclosure, the area security of the detection area is improved. Specifically, the reason why the area security of the detection area is low is that unknown unmanned aerial vehicles may be illegally modified (for example, signal anti-interference modules are added), and electromagnetic interference cannot be used for effective unmanned aerial vehicle control, resulting in low area security of the detection area. Based on this, the unmanned aerial vehicle control method based on unmanned aerial vehicle vision of some embodiments of the present disclosure first controls the signal emitting device to emit signals in real time to monitor unregistered unmanned aerial vehicles. In this way, it can be detected in real time whether there are unknown unmanned aerial vehicles in the detection area. Second, in response to receiving a reflection signal representing that an unknown flying object is monitored, the identification device is controlled to perform identification processing on the unknown flying object to generate an identification result. Then, in response to the identification result representing that the unknown flying object is an unknown unmanned aerial vehicle, the monitoring unmanned aerial vehicle in the unmanned aerial vehicle group is controlled to perform image tracking processing on the unknown unmanned aerial vehicle to generate an unknown unmanned aerial vehicle image sequence. In this way, the unknown unmanned aerial vehicle can be tracked and photographed by the associated unmanned aerial vehicle, and clearer images of the unknown unmanned aerial vehicle can be obtained. Then, according to the reflection signal received in real time, the unknown unmanned aerial vehicle is positioned to generate a first unknown unmanned aerial vehicle trajectory; according to the positioning of the monitoring unmanned aerial vehicle and the unknown unmanned aerial vehicle image sequence, a second unknown unmanned aerial vehicle trajectory is generated, and the first unknown unmanned aerial vehicle trajectory is corrected according to the second unknown unmanned aerial vehicle trajectory to obtain a corrected unmanned aerial vehicle trajectory. In this way, the corrected unmanned aerial vehicle flight trajectory can be determined. Then, according to the corrected unmanned aerial vehicle trajectory, the unknown unmanned aerial vehicle is trajectory predicted to generate an unmanned aerial vehicle predicted trajectory. In this way, the unknown unmanned aerial vehicle can be trajectory predicted through the flight trajectory of the unknown unmanned aerial vehicle. Finally, according to the unmanned aerial vehicle predicted trajectory, the unmanned aerial vehicle group or the interception device is controlled to intercept the unknown unmanned aerial vehicle. In this way, the unknown unmanned aerial vehicle can be intercepted, and thus the area security of the detection area can be improved.

[0091] Further reference Figure 2 , as an implementation of the method shown in the above figures, the present disclosure provides some embodiments of an unmanned aerial vehicle control device based on unmanned aerial vehicle vision, which device embodiments correspond to those method embodiments shown in Figure 1 , and the unmanned aerial vehicle control device based on unmanned aerial vehicle vision can be applied to various electronic devices.

[0092] As Figure 2As shown, the UAV control device 200 based on UAV vision of some embodiments comprises a first control unit 201, a second control unit 202, a third control unit 203, a positioning unit 204, a generating unit 205, a trajectory prediction unit 206 and a fourth control unit 207. Among them, the first control unit 201 is configured to control the signal emitting device to emit signals in real time to monitor unregistered UAVs; the second control unit 202 is configured to control the identification device to perform identification processing on the unknown flying object to generate an identification result in response to receiving a reflected signal representing that an unknown flying object is monitored; the third control unit 203 is configured to control the monitoring UAV in the UAV group to perform image tracking processing on the unknown UAV to generate an unknown UAV image sequence in response to the identification result representing that the unknown flying object is an unknown UAV; the positioning unit 204 is configured to perform positioning processing on the unknown UAV according to the real-time received reflected signal to generate a first unknown UAV trajectory; the generating unit 205 is configured to generate a second unknown UAV trajectory according to the positioning of the monitoring UAV and the unknown UAV image sequence, and correct the first unknown UAV trajectory according to the second unknown UAV trajectory to obtain a corrected UAV trajectory; the trajectory prediction unit 206 is configured to perform trajectory prediction processing on the unknown UAV according to the corrected UAV trajectory to generate a UAV predicted trajectory; and the fourth control unit 207 is configured to control the UAV group or the interception device to perform interception processing on the unknown UAV according to the UAV predicted trajectory.

[0093] It can be understood that the units described in the UAV control device 200 based on UAV vision correspond to the respective steps in the method described above. Figure 1 The operations, features and advantages described above for the method also apply to the UAV control device 200 based on UAV vision and the units contained therein, which will not be described here.

[0094] Reference is made below to Figure 3 which shows a structural schematic diagram of an electronic device 300 suitable for implementing some embodiments of the present disclosure. The electronic device in some embodiments of the present disclosure can include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Tablet PCs), PMPs (Portable Multimedia Players), vehicle terminals (such as vehicle navigation terminals), and the like, as well as fixed terminals such as digital TVs, desktop computers, and the like. Figure 3 The electronic device shown is only an example and should not impose any limitation on the functions and use range of the embodiments of the present disclosure.

[0095] As Figure 3As shown, the electronic device 300 can include a processing device 301 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 302 or loaded into a random access memory (RAM) 303 from a storage device 308. Various programs and data required for the operation of the electronic device 300 are also stored in the RAM 303. The processing device 301, the ROM 302, and the RAM 303 are connected to each other through a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0096] Generally, the following devices can be connected to the I / O interface 305: input devices 306 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 308 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 309. The communication devices 309 can allow the electronic device 300 to communicate wirelessly or wired with other devices to exchange data. Although Figure 3 The electronic device 300 is shown with various devices, but it should be understood that all of the illustrated devices are not required, and fewer or different devices can alternatively be implemented. Figure 3 Each block shown in the flowcharts can represent a device, or multiple devices, as necessary.

[0097] In particular, processes described above with reference to the flowcharts can be implemented as a computer software program according to some embodiments of the present disclosure. For example, some embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program code for performing the methods illustrated by the flowcharts. In some such embodiments, the computer program can be downloaded and installed from a network through the communication devices 309, or installed from the storage devices 308, or installed from the ROM 302. When the computer program is executed by the processing device 301, the above-described functions defined in the methods of some embodiments of the present disclosure are performed.

[0098] Note that the computer readable medium in some embodiments of the present disclosure can be a computer readable signal medium or a computer readable storage medium or any combination thereof. The computer readable storage medium can be, for example but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any suitable combination of the foregoing. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In some embodiments of the present disclosure, the computer readable storage medium can be any tangible medium that contains or stores a program used by an instruction execution system, apparatus or device, or that can be used by or in connection with an instruction execution system, apparatus or device. In some embodiments of the present disclosure, the computer readable signal medium can include a computer readable program code propagated in or on a carrier medium, in which the computer readable program code is embodied. Such propagated computer readable program code can take many forms, including but not limited to, an electromagnetic signal, an optical signal or any suitable combination of the foregoing. The computer readable signal medium can also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate or transport a program for use by or in connection with an instruction execution system, apparatus or device. Program code embodied on a computer readable medium can be transmitted using any suitable medium, including but not limited to, wire, cable, wireless, RF, infrared or any suitable combination of the foregoing.

[0099] In some embodiments, the client, server, or both can communicate using any current known or future developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks ("LAN"), wide area networks ("WAN"), the Internet, and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any current known or future developed networks.

[0100] The computer readable medium can be included in the electronic device, or exist separately from the electronic device. The computer readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to control the signal emitting device to emit signals in real time to monitor an unregistered unmanned aerial vehicle. In response to receiving a reflected signal representing that an unknown flying object is monitored, the identification device is controlled to perform identification processing on the unknown flying object to generate an identification result. In response to the identification result representing that the unknown flying object is an unknown unmanned aerial vehicle, the monitoring unmanned aerial vehicle in the group of unmanned aerial vehicles is controlled to perform image tracking processing on the unknown unmanned aerial vehicle to generate an unknown unmanned aerial vehicle image sequence. The unknown unmanned aerial vehicle is positioned according to the reflected signal received in real time to generate a first unknown unmanned aerial vehicle trajectory. A second unknown unmanned aerial vehicle trajectory is generated according to the position of the monitoring unmanned aerial vehicle and the unknown unmanned aerial vehicle image sequence, and the first unknown unmanned aerial vehicle trajectory is corrected according to the second unknown unmanned aerial vehicle trajectory to obtain a corrected unmanned aerial vehicle trajectory. The unknown unmanned aerial vehicle is trajectory predicted according to the corrected unmanned aerial vehicle trajectory to generate an unmanned aerial vehicle predicted trajectory. The group of unmanned aerial vehicles or the interception device is controlled according to the unmanned aerial vehicle predicted trajectory to perform interception processing on the unknown unmanned aerial vehicle.

[0101] Computer program code for carrying out operations of some embodiments of the disclosure can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0102] The computer program product of the first aspect can include a computer readable storage medium. The computer readable storage medium can include instructions. The instructions can include one or both of: instructions for causing a computer to implement a method as described above; and instructions for causing a computer to operate based on a system as described above. The computer readable storage medium can include one or more of: a magnetic disk; a magnetic tape; a magneto-optical disk; a semiconductor memory (e.g., a RAM, a ROM, a flash memory, etc.); and an optical disk.

[0103] The units described in some embodiments of the present disclosure can be implemented by means of software, or by means of hardware. The described units can also be provided in a processor, for example, a processor can be described as comprising a first control unit, a second control unit, a third control unit, a positioning unit, a generating unit, a trajectory prediction unit and a fourth control unit. In some cases, the names of these units do not constitute a limitation on the units themselves, for example, the first control unit can also be described as a unit that controls the above-mentioned signal emitting device to emit signals in real time to monitor unregistered drones.

[0104] The functions described above can be performed, at least in part, by one or more hardware logic components. For example, and without limitation, example types of hardware logic components that can be used include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip systems (SOCs), complex programmable logic devices (CPLDs), etc.

[0105] The above description is merely exemplary of some preferred embodiments of the present disclosure and of the application of the principles thereof, and the scope of protection that is sought for the present disclosure is not limited to the specific embodiments described herein. It is, therefore, expressly intended that those alternatives, modifications, and variations of the present disclosure hereinbefore set forth, including, but not limited to, any alternative embodiments as to the theory and principle of the application, be fully within the scope of the following claims. For example, any features of the above described embodiments of the present disclosure, and any features of the above described embodiments of the present disclosure, can be interchanged and / or removed and / or rearranged, and should be within the scope of the present disclosure.

Claims

1. A method for controlling unmanned aerial vehicles (UAVs) based on UAV vision, applied to a UAV control system, the UAV control system comprising a signal emitting device, a UAV group and an identification device, the method comprising: controlling the signal emitting device to emit signals in real time to monitor unregistered UAVs; in response to receiving a reflected signal representing that an unknown flying object is monitored, controlling the identification device to perform identification processing on the unknown flying object to generate an identification result; in response to the identification result representing that the unknown flying object is an unknown UAV, controlling a monitoring UAV in the UAV group to perform image tracking processing on the unknown UAV to generate an unknown UAV image sequence; performing positioning processing on the unknown UAV according to real-time received reflected signals to generate a first unknown UAV trajectory; wherein the positioning processing on the unknown UAV according to the real-time received reflected signals to generate the first unknown UAV trajectory comprises: obtaining a preset real scene three-dimensional model, wherein the preset real scene three-dimensional model is a three-dimensional model of a signal emitting range corresponding to the signal emitting device; selecting at least one reflected signal from each of the real-time received reflected signals according to a preset time interval to obtain a reflected signal sequence; for each of the reflected signal sequence, marking a position represented by the reflected signal in the preset real scene three-dimensional model; generating a first unknown UAV trajectory according to the marked positions in the preset real scene three-dimensional model; generating a second unknown UAV trajectory according to the positioning of the monitoring UAV and the unknown UAV image sequence, and performing correction processing on the first unknown UAV trajectory according to the second unknown UAV trajectory to obtain a corrected UAV trajectory; performing trajectory prediction processing on the unknown UAV according to the corrected UAV trajectory to generate a UAV predicted trajectory; wherein the trajectory prediction processing on the unknown UAV according to the corrected UAV trajectory to generate the UAV predicted trajectory comprises: determining a UAV trajectory point set according to the corrected UAV trajectory; for each UAV trajectory point in the UAV trajectory point set, determining a UAV speed and a UAV displacement corresponding to the UAV trajectory point; generating a UAV speed sequence and a UAV displacement sequence according to the determined UAV speeds and UAV displacements; performing normalization processing on the UAV speed sequence and the UAV displacement sequence to generate a normalized UAV speed sequence and a normalized UAV displacement sequence; inputting the normalized UAV speed sequence and the normalized UAV displacement sequence into a pre-trained UAV trajectory prediction model to obtain a UAV predicted trajectory; controlling the UAV group or an interception device to perform interception processing on the unknown UAV according to the UAV predicted trajectory.

2. The method of claim 1, wherein, The controlling the identification device to perform identification processing on the unknown flying object to generate an identification result in response to receiving a reflected signal representing that an unknown flying object is monitored comprises: controlling the identification device to perform photographing processing on the unknown flying object to generate a photographing image sequence. input each of the photographed image sequences into a pre-trained unmanned aerial vehicle recognition model to obtain a recognition result.

3. The method of claim 1, wherein, The unmanned aerial vehicle trajectory prediction model is trained through the following steps: obtain a sample set, wherein a sample in the sample set includes a sample normalized unmanned aerial vehicle speed sequence and a sample normalized unmanned aerial vehicle displacement sequence, and a sample unmanned aerial vehicle predicted trajectory corresponding to the sample normalized unmanned aerial vehicle speed sequence and the sample normalized unmanned aerial vehicle displacement sequence; select a sample from the sample set; input the sample into an initial network model to obtain an unmanned aerial vehicle predicted trajectory corresponding to the sample; determine a loss value between the unmanned aerial vehicle predicted trajectory corresponding to the sample and the sample unmanned aerial vehicle predicted trajectory included in the sample; in response to the loss value being greater than or equal to a preset threshold, adjust network parameters of the initial network model.

4. The method of claim 3, wherein, The method further includes: in response to the loss value being less than the preset threshold, determining the initial network model as the unmanned aerial vehicle trajectory prediction model.

5. The method of claim 1, wherein, The unmanned aerial vehicle control method further includes an interception device; and controlling the unmanned aerial vehicle group or the interception device to intercept the unknown unmanned aerial vehicle according to the unmanned aerial vehicle predicted trajectory includes: performing size prediction on the unknown unmanned aerial vehicle according to the unknown unmanned aerial vehicle image sequence to obtain a size prediction result; determining whether the unmanned aerial vehicle predicted trajectory satisfies a preset trajectory condition; in response to the unmanned aerial vehicle predicted trajectory satisfying the preset trajectory condition, controlling an interception device included in the unmanned aerial vehicle control system to perform a catapult interception operation on the unknown unmanned aerial vehicle according to the unmanned aerial vehicle predicted trajectory; in response to the unmanned aerial vehicle predicted trajectory not satisfying the preset trajectory condition, controlling the unmanned aerial vehicle group to perform a dynamic interception operation on the unknown unmanned aerial vehicle according to the size prediction result and the unmanned aerial vehicle predicted trajectory.

6. An unmanned aerial vehicle control device based on unmanned aerial vehicle vision, configured to perform the unmanned aerial vehicle control method according to any one of claims 1-5, and including: a first control unit configured to control a signal emission device to emit signals in real time to monitor unregistered unmanned aerial vehicles; a second control unit configured to, in response to receiving a reflection signal representing that an unknown flying object is monitored, control an identification device to perform identification processing on the unknown flying object to generate an identification result; a third control unit configured to, in response to the identification result representing that the unknown flying object is an unknown unmanned aerial vehicle, control a monitoring unmanned aerial vehicle in an unmanned aerial vehicle group to perform image tracking processing on the unknown unmanned aerial vehicle to generate an unknown unmanned aerial vehicle image sequence; a positioning unit configured to perform positioning processing on the unknown unmanned aerial vehicle according to reflection signals received in real time to generate a first unknown unmanned aerial vehicle trajectory; the positioning unit is further configured to: obtain a preset real scene three-dimensional model, wherein the preset real scene three-dimensional model is a three-dimensional model of a signal emission range corresponding to the signal emission device; select at least one reflection signal from each reflection signal received in real time at a preset time interval to obtain a reflection signal sequence; For each reflection signal in the reflection signal sequence, a position represented by the reflection signal is labeled in the preset real scene three-dimensional model; According to the positions labeled in the preset real scene three-dimensional model, a first unknown unmanned aerial vehicle trajectory is generated; A generating unit is configured to generate a second unknown unmanned aerial vehicle trajectory according to the positioning of the monitoring unmanned aerial vehicle and the unknown unmanned aerial vehicle image sequence, and to correct the first unknown unmanned aerial vehicle trajectory according to the second unknown unmanned aerial vehicle trajectory to obtain a corrected unmanned aerial vehicle trajectory; A trajectory prediction unit is configured to perform trajectory prediction processing on the unknown unmanned aerial vehicle according to the corrected unmanned aerial vehicle trajectory to generate an unmanned aerial vehicle predicted trajectory; the trajectory prediction unit is further configured to: According to the corrected unmanned aerial vehicle trajectory, a set of unmanned aerial vehicle trajectory points is determined; For each unmanned aerial vehicle trajectory point in the set of unmanned aerial vehicle trajectory points, a corresponding unmanned aerial vehicle speed and unmanned aerial vehicle displacement of the unmanned aerial vehicle trajectory point are determined; According to the determined respective unmanned aerial vehicle speeds and respective unmanned aerial vehicle displacements, an unmanned aerial vehicle speed sequence and an unmanned aerial vehicle displacement sequence are generated; The unmanned aerial vehicle speed sequence and the unmanned aerial vehicle displacement sequence are normalized to generate a normalized unmanned aerial vehicle speed sequence and a normalized unmanned aerial vehicle displacement sequence; The normalized unmanned aerial vehicle speed sequence and the normalized unmanned aerial vehicle displacement sequence are input into a pre-trained unmanned aerial vehicle trajectory prediction model to obtain an unmanned aerial vehicle predicted trajectory; A fourth control unit is configured to control the unmanned aerial vehicle group or an interception device to perform interception processing on the unknown unmanned aerial vehicle according to the unmanned aerial vehicle predicted trajectory.

7. An electronic device, comprising: one or more processors; a storage device having one or more programs stored thereon; when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1-5.

8. A computer readable medium having stored thereon a computer program, wherein, The program is executed by the processor to implement the method of any one of claims 1-5.

Images