Unmanned aerial vehicle real-time identification and tracking system based on neural network

Abstract

The invention discloses an unmanned aerial vehicle real-time identification and tracking system based on a neural network, and the system carries out the identification and tracking of a target unmanned aerial vehicle through the fusion of a millimeter-wave radar and a network camera, i.e., obtains a three-dimensional target coordinate through the millimeter-wave radar, achieves the all-weather monitoring of the target unmanned aerial vehicle, and achieves the real-time identification and tracking of the target unmanned aerial vehicle through the combination of an image recognition module of the network camera. The method achieves the precise recognition and confirmation of the type and position of a target, and improves the robustness and recognition accuracy in a complex environment. A closed-loop multi-strategy holder control algorithm is adopted, a tracking control module of the server is fused with three control strategies of proportion, fuzzy and integral differential adjustment, a holder control instruction is calculated in real time according to the target state, accurate turning and continuous tracking of a camera are achieved, the stability of the target unmanned aerial vehicle kept in the center of a picture is improved, and the target unmanned aerial vehicle is more stable. The requirements of continuous, stable and closed-loop tracking of the unmanned aerial vehicle in a complex dynamic environment are met, and the system has the technical advantages of being high in adaptive capacity, high in response speed and high in recognition precision.

Description

TECHNICAL FIELD

[0001] The embodiment of the application relates to the technical field of unmanned aerial vehicles, in particular to a neural network-based real-time identification and tracking system for unmanned aerial vehicles. BACKGROUND

[0002] With the rapid development of low-altitude unmanned aerial vehicle technology, the unmanned aerial vehicles are increasingly widely applied in the fields of logistics transportation, aerial inspection, emergency response and entertainment aerial photography. However, the problems of privacy security, flight control and airspace order caused by the frequent activities of unmanned aerial vehicles in urban airspace are increasingly prominent. Therefore, the real-time identification and accurate tracking of low-altitude unmanned aerial vehicles have become an important research direction in the field of airspace safety.

[0003] The existing unmanned aerial vehicle identification and tracking system mainly relies on a single sensor, such as an optical camera for image identification or a radar system for target detection. However, the single sensor system has the following limitations:

[0004] (1) The optical camera has low identification accuracy in the night, strong light interference or shielding environment;

[0005] (2) Although the radar system has all-weather working capability, it is difficult to accurately locate the specific position of the target in the image;

[0006] (3) Lack of fusion processing strategy, leading to difficulty in balancing identification accuracy and tracking stability.

[0007] In addition, the existing unmanned aerial vehicle identification and tracking system mostly adopts a serial control strategy, which has the problems of insufficient algorithm robustness and low gimbal control accuracy, and is difficult to meet the needs of continuous, stable and closed-loop tracking of unmanned aerial vehicles in complex dynamic environments. Therefore, it is an urgent technical problem to develop a real-time identification and tracking system for unmanned aerial vehicles which can accurately identify and stably track low-altitude unmanned aerial vehicles. SUMMARY

[0008] The embodiment of the application provides a neural network-based real-time identification and tracking system for unmanned aerial vehicles, which fuses millimeter wave radar and network camera perception data, and combines deep learning image identification and fusion processing strategy, and realizes high-precision identification and stable tracking of low-altitude unmanned aerial vehicles.

[0009] In the first aspect, the embodiment of the application provides a neural network-based real-time identification and tracking system for unmanned aerial vehicles, which comprises:

[0010] A plurality of sensors, the sensors comprising a millimeter wave radar, a network camera, a microprocessor and a first wireless transceiver module, the millimeter wave radar being connected to the microprocessor through a wired protocol, the microprocessor being connected to the first wireless transceiver module through a wired protocol, the network camera being connected to the first wireless transceiver module through a wired protocol, the millimeter wave radar being used to acquire three-dimensional spatial coordinate information of a target unmanned aerial vehicle, the network camera being used to acquire real-time image information of the target unmanned aerial vehicle, the network camera having an image recognition module, the image recognition module adopting a deep neural network structure, the image recognition module performing feature extraction and multi-scale detection on the real-time image information, and outputting a category and two-dimensional image coordinates of a target;

[0011] A server, the server being wirelessly connected to the first wireless transceiver module, the server having a tracking control module, the tracking control module adopting a weighted fusion processing strategy based on a proportional control strategy, a fuzzy control strategy and an integral differential adjustment control strategy, the tracking control module calculating a gimbal angle change amount based on the weighted fusion processing strategy according to the three-dimensional spatial coordinate information and the two-dimensional image coordinate information, generating a gimbal control instruction according to the gimbal angle change amount, and outputting the gimbal control instruction to control the network camera to rotate or zoom, so that the network camera continuously tracks and photographs the target unmanned aerial vehicle;

[0012] A client, the client comprising a second wireless transceiver module and a monitoring terminal, the second wireless transceiver module being wirelessly connected to the server, the target unmanned aerial vehicle being kept centered on a monitoring interface of the monitoring terminal through adjustment of a view angle of the network camera.

[0013] In an embodiment, the millimeter wave radar pre-selects a target echo point, calculates a confidence degree according to amplitude reflection intensity, distance and inter-frame speed estimation, projects the selected target echo point set to an image two-dimensional coordinate point set, and dynamically sets a region of interest according to a dispersion degree of the image two-dimensional coordinate point set.

[0014] In an embodiment, the server generates a target trajectory curve based on the target echo point set, obtains a planar projection coordinate through coordinate transformation, constructs an adaptive region of interest with a projection coordinate center, and controls a field of view of the network camera to move to the region of interest based on target offset and confidence degree fluctuation.

[0015] In an embodiment, the deep neural network is composed of a backbone network, a neck network and a detection head; the backbone network outputs three feature maps through the neck network for use by the detection head, an SPPF layer is used in the last layer of the backbone network, and the SPPF layer extracts multi-scale context information through multiple max-pooling layers; the neck network includes an up-sampling layer and a concatenation layer, the up-sampling layer adopts a nearest neighbor interpolation up-sampling manner, and the concatenation layer is used to connect feature maps of multiple scales.

[0016] In an embodiment, the image recognition module uses non-maximum suppression to eliminate redundant overlapping boxes, for multiple detection boxes of the same object, selects the detection box with the highest score as the target box, and removes all detection boxes with an intersection-over-union greater than a threshold value with the target box.

[0017] In an embodiment, the microprocessor pre-processes and filters the information returned by the millimeter wave radar, eliminates static interference points, and extracts the spatial position of the suspected target drone; the microprocessor determines whether the current target drone deviates from the center of the image according to the offset of the center of the target box from the center of the image, and if the target drone tracking is lost, controls the millimeter wave radar to re-perform detection and image recognition.

[0018] In an embodiment, the proportional control strategy is used to quickly respond to the instantaneous offset of the target position in the image; the fuzzy control strategy is used for flexible adjustment when the offset is less than a preset threshold value, to suppress the false action caused by image jitter; and the integral and differential adjustment strategy is used to compensate for the slow accumulation of small offset trends and accelerate the response in the case of severe offset.

[0019] In an embodiment, after the tracking control module receives the recognition box coordinates, it calculates the fusion coefficient and the control speed, the fused control speed instruction is packaged in a standardized JSON format, and is sent to a control service interface built based on a Flask framework through an HTTP POST request; after the control service interface receives the control instruction, it is translated into a continuous action command conforming to the ONVIF standard protocol, and real-time pan-tilt adjustment is realized through the built-in interface of the network camera; after the pan-tilt action is executed, the network camera continuously returns images, and the system performs a new round of recognition, judgment and control instruction generation on each frame of image, forming a closed-loop tracking.

[0020] In an embodiment, the tracking control module adopts a confidence sliding window fusion strategy in each control cycle:

[0021]

[0022] wherein, is the current confidence smoothing value, C iThe confidence degree of the i-th frame detection result is N, the window length, and the confidence degree smoothing value is used to determine whether the target continuously exists or the confidence degree continuously decreases, and if it is considered that the target may be lost or unstable in recognition, and a search or re-identification mode is entered; if it is considered that the target state is stable, and the current tracking strategy is maintained, wherein τ min , τ stable are configurable thresholds.

[0023] In an embodiment, the system adopts an RTSP+HTTP wireless communication mode, supports multi-thread asynchronous processing, multi-user concurrent access, and human-computer interaction of a visual control interface.

[0024] The embodiment of the present application comprises: a neural network-based unmanned aerial vehicle real-time identification tracking system, which comprises a plurality of sensors, a server and a client, wherein the sensors comprise a millimeter wave radar, a network camera, a microprocessor and a first wireless transceiver module, the millimeter wave radar is connected to the microprocessor through a wired protocol, the microprocessor is connected to the first wireless transceiver module through a wired protocol, the network camera is connected to the first wireless transceiver module through a wired protocol, the millimeter wave radar is used to obtain three-dimensional space coordinate information of a target unmanned aerial vehicle, the network camera is used to obtain real-time image information of the target unmanned aerial vehicle, the network camera has an image recognition module, the image recognition module adopts a deep neural network structure, the image recognition module performs feature extraction and multi-scale detection on the real-time image information, and outputs the category and two-dimensional image coordinates of the target; the server is wirelessly connected to the first wireless transceiver module, the server has a tracking control module, the tracking control module adopts a weighted fusion processing strategy based on proportional control strategy, fuzzy control strategy and integral differential adjustment control strategy, the tracking control module calculates a change amount of a holder angle based on the weighted fusion processing strategy according to the three-dimensional space coordinate information and the two-dimensional image coordinate information, generates a holder control instruction according to the change amount of the holder angle, and outputs the holder control instruction to control the network camera to rotate or zoom, so that the network camera continuously tracks and photographs the target unmanned aerial vehicle; the client comprises a second wireless transceiver module and a monitoring terminal, the second wireless transceiver module is wirelessly connected to the server, and the target unmanned aerial vehicle is kept centered on the monitoring interface of the monitoring terminal through network camera view angle adjustment. Based on this, the embodiment of the present application fuses the millimeter wave radar and the network camera to identify and track the target unmanned aerial vehicle, that is, the three-dimensional target coordinates are obtained through the millimeter wave radar, all-weather monitoring of the target unmanned aerial vehicle is realized, the image recognition module of the network camera is combined, accurate identification and confirmation of the target category and position are realized, and the robustness and identification accuracy in a complex environment are enhanced. Moreover, the embodiment of the present application adopts a closed-loop multi-strategy holder control algorithm, the tracking control module of the server fuses three control strategies of proportion, fuzzy and integral differential adjustment, the holder control instruction is calculated in real time according to the target state, accurate rotation and continuous tracking of the camera are realized, the stability of the target unmanned aerial vehicle kept in the center of the picture is improved, the demand for continuous, stable and closed-loop tracking of the unmanned aerial vehicle in a complex dynamic environment is met, and based on this, the neural network-based unmanned aerial vehicle real-time identification tracking system of the embodiment of the present application has the technical advantages of strong self-adaptability, fast response speed and high identification accuracy.

[0025] Other features and advantages of the present application will be set forth in the following description, and in part will become apparent to those skilled in the art from the description, or can be learned by practice of the present application. The objects and other advantages of the present application will be realized and achieved by means of the instrumentalities and combinations pointed out in the description and appended claims. BRIEF DESCRIPTION OF DRAWINGS

[0026] The accompanying drawings are used to provide further understanding of the technical solutions of the present application and constitute a part of the specification, and are used to explain the technical solutions of the present application together with embodiments of the present application, and do not constitute a limitation on the technical solutions of the present application.

[0027] Figure 1 A schematic diagram of the overall structure of a neural network-based unmanned aerial vehicle real-time identification and tracking system according to an embodiment of the present application is shown in FIG. 1.

[0028] Figure 2 A schematic diagram of the information flow and instruction flow of an unmanned aerial vehicle identification and tracking system according to an embodiment of the present application is shown in FIG. 2.

[0029] Figure 3 A flowchart of target identification and control using millimeter wave and image fusion according to an embodiment of the present application is shown in FIG. 3.

[0030] Figure 4 A logic diagram of deep neural network image identification according to an embodiment of the present application is shown in FIG. 4.

[0031] Figure 5 A logic diagram of gimbal tracking control algorithm of a tracking control module according to an embodiment of the present application is shown in FIG. 5.

[0032] Figure 6 A schematic diagram of a feasible radar deployment scheme according to an embodiment of the present application is shown in FIG. 6. DETAILED DESCRIPTION

[0033] In order to make the objects, technical solutions, and advantages of the present application clearer, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0034] It should be understood that, in the description of the embodiments of the present application, the meaning of multiple (or multiple items) is more than two, greater than, less than, more than, and the like are not included in the number, and above, below, and the like are included in the number. If there is a description of "first", "second", and the like, it is only used for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features or the sequence of indicated technical features.

[0035] The existing unmanned aerial vehicle identification and tracking system mainly relies on a single sensor, such as using an optical camera for image identification, or using a radar system for target detection. However, the single sensor system has the following limitations:

[0036] (1) The optical camera has low recognition accuracy at night, in strong light interference, or in a blocked environment;

[0037] (2) The radar system has all-weather working ability, but it is difficult to accurately locate the specific position of the target in the image;

[0038] (3) Lack of fusion processing strategy, leading to difficulty in balancing recognition accuracy and tracking stability.

[0039] In addition, the existing unmanned aerial vehicle recognition and tracking system mostly adopts a serial control strategy, which has problems such as insufficient algorithm robustness and low gimbal control accuracy, and is difficult to meet the demand for continuous, stable and closed-loop tracking of unmanned aerial vehicles in complex dynamic environments. Therefore, developing a real-time unmanned aerial vehicle recognition and tracking system capable of high-precision recognition and stable tracking of low-altitude unmanned aerial vehicles has become a technical problem to be solved.

[0040] In view of the above problems in the prior art, the embodiment of the present application provides a neural network-based real-time identification and tracking system for unmanned aerial vehicles, which comprises a plurality of sensors, a server and a client, wherein the sensors comprise a millimeter wave radar, a network camera, a microprocessor and a first wireless transceiver module, the millimeter wave radar is connected to the microprocessor through a wired protocol, the microprocessor is connected to the first wireless transceiver module through a wired protocol, the network camera is connected to the first wireless transceiver module through a wired protocol, the millimeter wave radar is used to acquire three-dimensional spatial coordinate information of a target unmanned aerial vehicle, and the network camera is used to acquire real-time image information of the target unmanned aerial vehicle; the network camera has an image recognition module, the image recognition module adopts a deep neural network structure, the image recognition module performs feature extraction and multi-scale detection on the real-time image information, and outputs the category and two-dimensional image coordinates of the target; the server is wirelessly connected to the first wireless transceiver module, the server has a tracking control module, the tracking control module adopts a weighted fusion processing strategy based on proportional control strategy, fuzzy control strategy and integral differential adjustment control strategy, the tracking control module calculates a change amount of a pan-tilt angle based on the weighted fusion processing strategy according to the three-dimensional spatial coordinate information and the two-dimensional image coordinate information, generates a pan-tilt control instruction according to the change amount of the pan-tilt angle, and outputs the pan-tilt control instruction to control the network camera to rotate or zoom, so that the network camera continuously tracks and photographs the target unmanned aerial vehicle; the client comprises a second wireless transceiver module and a monitoring terminal, the second wireless transceiver module is wirelessly connected to the server, and the target unmanned aerial vehicle is kept centered on the monitoring interface of the monitoring terminal through adjustment of the field of view of the network camera. Based on this, the embodiment of the present application fuses the millimeter wave radar and the network camera to identify and track the target unmanned aerial vehicle, that is, the three-dimensional target coordinates are acquired through the millimeter wave radar to realize all-weather monitoring of the target unmanned aerial vehicle, and the image recognition module of the network camera is combined to realize accurate identification and confirmation of the target category and position, thereby enhancing the robustness and identification accuracy in a complex environment. Moreover, the embodiment of the present application adopts a closed-loop multi-strategy pan-tilt control algorithm, three control strategies of proportional control, fuzzy control and integral differential adjustment control are fused by the tracking control module of the server, the pan-tilt control instruction is calculated in real time according to the target state, accurate rotation and continuous tracking of the camera are realized, the stability of the target unmanned aerial vehicle kept in the center of the picture is improved, the demand for continuous, stable and closed-loop tracking of the unmanned aerial vehicle in a complex dynamic environment is met, and based on this, the neural network-based real-time identification and tracking system for unmanned aerial vehicles has the technical advantages of strong self-adaptability, fast response speed and high identification accuracy.

[0041] It can be understood that, in order to identify and locate the unmanned aerial vehicle target, the system is equipped with a millimeter wave radar for active perception of flying objects in the airspace. The millimeter wave radar is installed on the fixed support of the system platform, has high detection sensitivity and long-distance positioning capability. The radar is connected to the microprocessor through a serial wired communication protocol, and periodically outputs the three-dimensional spatial coordinate information of the detected flying objects, including horizontal angle, pitch angle and distance, etc. The microprocessor pre-processes and filters the information returned by the radar, eliminates static interference points, and extracts the spatial position of the suspected unmanned aerial vehicle target as the region of interest for the tracking control module.

[0042] It can be understood that, for the network camera, the system uses a high-definition network camera supporting the ONVIF protocol, which has pan-tilt control capability and can accurately rotate in the horizontal and vertical directions to realize visual coverage of the specified airspace area. The network camera is connected to the microprocessor through wired Ethernet, supporting stable and low-latency video transmission. After receiving the radar coordinate information from the microprocessor, the network camera adjusts the direction of the pan-tilt according to the approximate direction of the target, so that the visual area is aligned with the target airspace, thereby providing clear image data for the image recognition module.

[0043] It can be understood that, for the image recognition module, a deep neural network is used for target recognition of image stream. The deep neural network identifies and classifies flying objects in the image, outputting bounding box, class (such as unmanned aerial vehicle or other interference object), and confidence score. To improve recognition stability, a multi-frame fusion strategy can also be used to effectively reduce false positives and missed detections caused by image blur or temporary occlusion, which can accurately identify different types of small flying vehicles in complex backgrounds and is suitable for various weather and lighting conditions.

[0044] It can be understood that after completing the identification of the unmanned aerial vehicle, the system runs the tracking control module, which adopts a multi-strategy control fusion, weights and outputs the control speed of the target unmanned aerial vehicle, and continuously tracks the identified target unmanned aerial vehicle. The tracking control module can output the position of the target in the image in real time, maintaining high frame rate and stability. During tracking, the microprocessor determines whether the current target deviates from the center of the image according to the offset between the center of the target box and the center of the image, and if the tracking target is lost, the millimeter wave radar detection and image recognition are re-executed. The system controls the rotation direction and speed of the pan-tilt through the ONVIF protocol according to the control speed output by the tracking control module, so that the target remains in the center area of the network camera field of view. This process constitutes a closed-loop control system based on image feedback, with high stability and response speed.

[0045] The application will be described in detail below with reference to the drawings.

[0046] As Figure 1 shown, Figure 1 is a schematic diagram of the overall structure of a neural network-based real-time unmanned aerial vehicle (UAV) identification and tracking system according to an embodiment of the application. In the diagram, 1-A is a millimeter wave radar, 1-B is a network camera, 1-C is a server, 1-D1 is a second wireless transceiver module, 1-D2 is a first wireless transceiver module, 1-E is a microprocessor, and 1-F is a client. The millimeter wave radar is connected to the microprocessor through a wired protocol, the microprocessor is connected to the first wireless transceiver module through a wired protocol, and the network camera is connected to the first wireless transceiver module through a wired protocol. The millimeter wave radar is configured to obtain three-dimensional spatial coordinate information of a target UAV, and the network camera is configured to obtain real-time image information of the target UAV. The network camera has an image recognition module, which adopts a deep neural network structure. The image recognition module performs feature extraction and multi-scale detection on the real-time image information and outputs the category and two-dimensional image coordinates of the target. The server is wirelessly connected to the first wireless transceiver module. The server has a tracking control module, which adopts a weighted fusion processing strategy based on proportional control strategy, fuzzy control strategy, and integral differential adjustment control strategy. Based on the weighted fusion processing strategy, the tracking control module calculates a pan-tilt angle change amount based on the three-dimensional spatial coordinate information and the two-dimensional image coordinate information, generates a pan-tilt control instruction based on the pan-tilt angle change amount, and outputs the pan-tilt control instruction to control the network camera to rotate or zoom, so that the network camera continuously tracks and photographs the target UAV. The client includes the second wireless transceiver module and a monitoring terminal, and the second wireless transceiver module is wirelessly connected to the server.

[0047] As Figure 2 shown, Figure 2 is a schematic diagram of the information flow and instruction flow of the UAV identification and tracking system, taking a ROS2 master node as an example. In the diagram, 2-A represents that the millimeter wave radar provides the three-dimensional coordinates of the UAV, 2-B represents that the image recognition module locates the two-dimensional position of the UAV in the image coordinates, 2-C represents that the ROS2 master node performs fusion analysis and issues a pan-tilt control instruction, 2-D represents a monitoring interface of the client, through which the target UAV is kept centered in the monitoring interface of the monitoring terminal by adjusting the viewing angle of the network camera, and 2-E represents that the identification and control results are fed back to the monitoring interface.

[0048] As Figure 3 shown, Figure 3Fig. 3 is a flow chart of target recognition and control process for millimeter wave and image fusion, wherein 3-A represents that millimeter wave radar performs confidence screening on all echo points and retains high confidence points; 3-B represents that the ROS2 master node generates a target trajectory curve based on the screened point set, and coordinate transformation is performed to obtain plane projection coordinates; 3-C represents that the ROS2 master node constructs an adaptive region of interest (ROI) with the center of the projection coordinates; and 3-D represents that based on target offset, confidence fluctuation and other characteristics, the field of view is moved to the region of interest (ROI).

[0049] As shown in Fig. 4, Figure 4 Figure 4 Fig. 4 is a logic diagram of neural network image recognition, wherein 4-A represents video stream input (RTSP), 4-B represents image inference processing flow, 4-C represents output of bounding box and category information, and 4-D represents drawing and outputting of recognition result, picture and numbered list.

[0050] As shown in Fig. 5, Figure 5 Figure 5 Fig. 5 is a logic diagram of pan-tilt tracking control module, wherein 5-A represents calculation of tracking fusion coefficient, 5-B represents generation of adjustment instruction (direction / speed), 5-C represents adjustment of camera view angle to keep the target in the center, and 5-D represents continuous closed-loop recognition feedback.

[0051] It can be understood that in the neural network-based real-time recognition and tracking system of the unmanned aerial vehicle, the millimeter wave radar (1-A) continuously scans the airspace to obtain three-dimensional spatial coordinate information (2-A) of the aerial target, and the output is usually represented as a point in the radar coordinate system R The spatial position and posture of each millimeter wave radar node are represented by a translation vector T WR and a rotation matrix R WR , which define the position and direction of the millimeter wave radar coordinate system relative to the unified world coordinate system W. In order to convert the multiple point target echoes obtained by the millimeter wave radar into a stable perception region in the image coordinate, the system introduces a radar echo point fusion image projection and ROI (Region of Interest) adaptive scheduling algorithm.

[0052] In a millimeter wave radar scanning period, the same aerial target usually generates multiple high-confidence scattering points. The system first performs pre-screening (3-A) according to the target confidence function:

[0053]

[0054] wherein A i is the amplitude reflection intensity, D i is the distance between the target and the radar, V i is the inter-frame velocity estimation, and λ1, λ2 and λ3 are empirical weighting coefficients. If C i >τ​​c If so, the point is considered as a valid target point reservation.

[0055] Filtered point set The point will be mapped from the millimeter wave radar coordinate system to the world coordinate system, and then the local geometric fitting is used to generate the target motion trajectory curve C(t), and the center point is calculated:

[0056]

[0057] The center point is transmitted to the image projection process as the representative position of the target. First, the network camera external parameter is used to convert the target point to the camera coordinate system:

[0058] Ρ C =R CW ·Ρ center +Τ CW

[0059] The camera intrinsic matrix K performs image projection under the pinhole model (3-B):

[0060]

[0061] The projection result (u, v) is the center point of the region of interest (ROI) in the image. The system further calculates the dispersion of the point set in each direction:

[0062]

[0063] Further, dynamically set the image ROI area width and height (3-C):

[0064] ω=k1·σ x , h=k2·σ y

[0065] Where k1, k2 are scheduling scaling coefficients, which support flexible adjustment of the recognition area range according to the target distribution characteristics.

[0066] After completing the image ROI area construction, the ROS2 master node generates camera rotation instructions based on the center coordinates (u, v) of the region of interest, which are used to guide the pre-adjustment of the holder (3-D). According to the offset between the image center and the ROI center point (u, v) Δx=u-x0, Δy=v-y0, a simple linear proportional control command is generated:

[0067] V pan =K x ·Δx, v tilt =K y ·Δy

[0068] The ROS2 master node calls the camera holder movement interface to complete the pre-adjustment of the viewing angle.

[0069] It can be understood that the image recognition module adopts a deep neural network structure, which is composed of a backbone network, a neck network and a detection head, as shown in Figure 4 As shown, the deep neural network performs feature extraction and multi-scale detection on the input image frame (4-B), and outputs the category and two-dimensional image coordinates of the target (4-C). The ROS2 master node combines the image coordinates and radar spatial coordinates to calculate the change in the pan-tilt angle, control the camera rotation or zoom, and keep the target in the center of the screen (5-C in the figure), so as to realize accurate closed-loop tracking.

[0070] It can be understood that the C3 layer of the backbone network of the deep neural network is used to realize multi-scale information fusion and enhanced expression ability in feature extraction, and the input channel is split into two parts, one part is directly input, and the other part enters the neck network structure, and finally the convolution is performed after the detection head is merged and then output. The CONV layer of the deep neural network is a convolution layer, which performs weighted summation on the image or feature map through a sliding convolution kernel, thereby changing the feature map dimension, extracting local structure and semantic information. The last layer of the backbone network is the SPPF layer, which extracts multi-scale context information through multiple max pooling layers to enhance the perception ability of the multi-scale model. In the neck network, the up-sampling layer and the splicing layer are the core components of the neck network, which are used to fuse feature maps of different scales and realize multi-scale target detection capability. The up-sampling layer uses nearest neighbor interpolation up-sampling method, which is to find the nearest pixel in the original image for each pixel point of the target image, and directly copy the color value of the pixel, which can quickly realize resolution enhancement. The splicing layer connects feature maps of multiple scales to enhance the recognition ability of targets of different sizes.

[0071] It can be understood that the communication mode of the system adopts RTSP+HTTP wireless communication, supports multi-thread asynchronous processing, multi-user concurrent access, and human-computer interaction of visual control interface. As shown in Figure 3 After the FLASK service interface receives the camera video stream (4-A), it first decodes through a background thread or process, and then transmits each frame of image into the image recognition module (4-B). After the inference is completed, the target frame coordinates and category information are returned (4-C). The image result and target number can be further broadcasted to the system control interface (2-D) or other node modules through the serial interface (4-D).

[0072] It can be understood that the tracking control module combines target position offset, confidence change, target behavior characteristics and other multi-dimensional information, and fuses proportional control, fuzzy adjustment, integral and differential adjustment and other control strategies to realize dynamic adjustment and stable tracking of the holder. The image captured by the network camera is processed by the image recognition module, and the left upper corner and right lower corner coordinates [x1, y1, x2, y2] of the target bounding box are output, and the target center point coordinates offset Δx = x c - x0, Δy = y c - y0.

[0073] The proportional control strategy is used to quickly respond to the instantaneous offset of the target position in the image, and is controlled by a linear proportional factor:

[0074]

[0075] wherein, and are the proportional gain coefficients in the horizontal and vertical directions, respectively, with the unit of angular velocity / pixel.

[0076] The fuzzy control strategy is a flexible adjustment when the offset is small, and suppresses the misoperation caused by image jitter. The fuzzy control strategy is based on a fuzzy logic rule base, and outputs the speed command by fuzzy reasoning on the offset amplitude |Δx|, |Δy| and the change rate (Δx t - Δx t-1 ).

[0077]

[0078] wherein, μ x , μ y is a fuzzy reasoning function that outputs the control result according to the fuzzy rules.

[0079] The integral and differential adjustment strategy is used to compensate for the slow accumulation of small offset trends and accelerate the response to severe offset, and its control expression is as follows:

[0080]

[0081] wherein, is the integral and differential coefficient in the horizontal direction, is the integral and differential coefficient in the vertical direction, T is the integral window length, and Δx t is the offset of the t-th frame.

[0082] The final output control speed command of the tracking control module is obtained by weighted fusion of the proportional control, fuzzy adjustment and integral and differential adjustment three control strategies, that is, multi-strategy control fusion:

[0083]

[0084] wherein ω1, ω2, ω3 ∈ [0, 1] are dynamic fusion coefficients, satisfying the normalization condition ω1+ ω2+ ω3=1. The weight factor is configured according to the current tracking state of the target.

[0085] It can be understood that after the tracking control module receives the bounding box coordinates, the fusion coefficients and the control speed are calculated (5-A), the fused control speed instruction is packaged in a standardized JSON format, and is sent to the control service interface built based on the Flask framework through an HTTP POST request (5-B). After the control service interface receives the control instruction, it is translated into a continuous action command conforming to the ONVIF standard protocol, and real-time pan-tilt adjustment is realized through the built-in interface of the network camera (5-C). After the pan-tilt action is executed, the network camera continuously returns images, and the system performs a new round of identification, judgment, and control instruction generation on each frame of image, forming a stable closed-loop tracking logic (5-D). To further improve the control robustness, the system also introduces a confidence sliding window fusion strategy in each control cycle:

[0086]

[0087] wherein, is the current confidence smoothing value, C i is the confidence of the i-th frame of detection result, and N is the window length (usually 3-5 frames). The confidence smoothing value is used to judge whether the target is "continuously present" or "confidence is continuously decreasing", if it is considered that the target may be lost or the identification is unstable, and the search or re-identification mode is entered. If it is considered that the target state is stable, and the current tracking strategy is maintained. Wherein τ min , τ stable are configurable thresholds (such as 0.3 and 0.6).

[0088] Figure 6 is a feasible embodiment of the present application, wherein 6-A1 represents a millimeter wave radar, 6-A2 represents a network camera arranged in an open field of view, 6-B represents a recognizable and trackable unmanned aerial vehicle target, 6-C represents a server, and 6-D represents a wireless transceiver module.

[0089] The system hardware is composed of a millimeter wave radar, a network camera, a microprocessor, a wireless communication device, and a server. The deployment scheme is as follows: Figure 6As shown, the unmanned aerial vehicle target (6-B) is within the detection range and the field of view range, and an area with a wide field of view is selected in the deployment site, and a millimeter wave radar (6-A1) is integrated, which has a detection distance of 2 kilometers and a horizontal angle of view of 120 degrees, and can output point cloud coordinate information of moving targets in real time within an error range of ±0.5 meters. The millimeter wave radar is connected to the microprocessor (model: ESP32-WROOM) through a serial port, the microprocessor receives and buffers the point cloud information, and after preliminary filtering, uploads the point cloud information to the upper computer through wireless means.

[0090] The network camera (6-A2) uses a high-definition network camera (model: Yushi TICA500) supporting the ONVIF protocol, which has an image resolution of 2K, supports pan-tilt control, 54x optical zoom, night infrared fill light and other functions. The network camera establishes a connection with the server (6-C) through the wireless transceiver module (6-D), and the image data is pushed to the server in the form of an RTSP stream.

[0091] It should be noted that the present embodiment only provides a feasible implementation scheme, and the number of radars, deployment scheme, protocol definition and hardware selection are not limited by the content described in the present embodiment.

[0092] Based on this, the present application proposes a neural network-based real-time unmanned aerial vehicle identification and tracking system, which fuses millimeter wave radar and network camera perception data, and combines deep learning image recognition and fusion processing strategy, to realize high-precision identification and stable tracking of low-altitude unmanned aerial vehicles. Compared with the prior art, the present application has at least the following beneficial effects:

[0093] 1. Multi-source sensor fusion improves perception reliability: three-dimensional target coordinates are obtained through the millimeter wave radar to realize all-weather monitoring; combined with the image recognition module, accurate identification and confirmation of target categories and positions are realized, and robustness and identification accuracy in complex environments are enhanced.

[0094] 2. Adaptive ROI scheduling algorithm based on spatial projection: the target motion trajectory is constructed using the radar point cloud and projected into the image coordinate space, and the image recognition area size and position are dynamically adjusted to improve the recognition speed and resource utilization efficiency.

[0095] 3. High-performance deep neural network recognition algorithm: a deep neural network structure with multi-scale feature fusion is adopted, which has the ability to quickly identify small targets in complex backgrounds and is suitable for various resolutions and occlusion situations.

[0096] 4. Closed-loop multi-strategy pan-tilt control algorithm: three control strategies of proportional, fuzzy and integral derivative adjustment are fused, and pan-tilt control instructions are calculated in real time according to the target state to realize accurate turning and continuous tracking of the network camera, and improve the stability of the target unmanned aerial vehicle remaining in the center of the picture.

[0097] 5. The hardware and software structure is flexible and easy to deploy: it supports the deployment of millimeter wave radar and network camera in different places, wireless communication, and has good field adaptability and scalability, and is suitable for various unmanned aerial vehicle monitoring scenes such as cities, airports and power inspection.

[0098] In summary, the unmanned aerial vehicle identification and tracking system based on the neural network realizes multi-sensor fusion and identification-tracking-control integration, and has self-adaptive adjustment and robust control capability, and can be widely applied to unmanned aerial vehicle supervision, low-altitude airspace security and the like, and has significant practical value and industrial prospect.

[0099] The above is a specific description of the preferred embodiment of the present application, but the present application is not limited to the above-mentioned embodiments, and those skilled in the art can make various equivalent modifications or replacements without departing from the spirit of the present application, and these equivalent modifications or replacements are all included in the scope defined by the claims of the present application.

Claims

1. A neural network-based real-time identification and tracking system for unmanned aerial vehicles, characterized by, The system comprises: A plurality of sensors, the sensors comprising a millimeter wave radar, a network camera, a microprocessor and a first wireless transceiver module, the millimeter wave radar being connected to the microprocessor through a wired protocol, the microprocessor being connected to the first wireless transceiver module through a wired protocol, the network camera being connected to the first wireless transceiver module through a wired protocol, the millimeter wave radar being used to acquire three-dimensional spatial coordinate information of a target unmanned aerial vehicle, the network camera being used to acquire real-time image information of the target unmanned aerial vehicle, the network camera having an image recognition module, the image recognition module adopting a deep neural network structure, the image recognition module performing feature extraction and multi-scale detection on the real-time image information and outputting a category of a target and two-dimensional image coordinates; A server wirelessly connected to the first wireless transceiver module, the server having a tracking control module, the tracking control module adopting a weighted fusion processing strategy based on proportional control strategy, fuzzy control strategy and integral differential adjustment control strategy, the tracking control module calculating a gimbal angle change amount based on the weighted fusion processing strategy according to the three-dimensional spatial coordinate information and the two-dimensional image coordinate information, generating a gimbal control instruction according to the gimbal angle change amount, and outputting the gimbal control instruction to control the network camera to rotate or zoom, so that the network camera continuously tracks and photographs the target unmanned aerial vehicle; A client comprising a second wireless transceiver module and a monitoring terminal, the second wireless transceiver module being wirelessly connected to the server, the target unmanned aerial vehicle being kept centered on a monitoring interface of the monitoring terminal through network camera view angle adjustment. 2.The real-time identification and tracking system based on neural network for UAVs according to claim 1, wherein, The millimeter wave radar pre-screens target echo points, calculates confidence based on amplitude reflection intensity, distance and inter-frame speed estimation, projects the screened target echo point set to an image two-dimensional coordinate point set, and dynamically sets a region of interest based on the dispersion degree of the image two-dimensional coordinate point set. 3.The real-time identification and tracking system based on neural network for UAVs according to claim 2, wherein, The server generates a target trajectory curve based on the target echo point set, obtains a planar projection coordinate through coordinate transformation, constructs an adaptive region of interest with the projection coordinate center, and controls the field of view of the network camera to move to the region of interest based on target offset and confidence fluctuation. 4.The real-time identification and tracking system for UAV based on neural network according to claim 1, wherein, The deep neural network is composed of a backbone network, a neck network and a detection head; the backbone network outputs three feature maps through the neck network for use by the detection head, an SPPF layer is used in the last layer of the backbone network, the SPPF layer extracts multi-scale context information through multiple max-pooling layers; the neck network comprises an up-sampling layer and a concatenation layer, the up-sampling layer adopts a nearest neighbor interpolation up-sampling method, and the concatenation layer is used to connect feature maps of multiple scales. 5.The real-time identification and tracking system for UAV based on neural network according to claim 1, wherein, The image recognition module uses non-maximum suppression to eliminate redundant overlapping boxes, for multiple detection boxes of the same object, selects the highest-scored detection box as the target box, and removes all detection boxes with an intersection-over-union greater than a threshold value with the target box. 6.The real-time identification and tracking system for UAV based on neural network according to claim 1, wherein, The microprocessor pre-processes and filters the information returned by the millimeter wave radar, eliminates static interference points, and extracts the spatial position of the suspected target unmanned aerial vehicle; the microprocessor judges whether the current target unmanned aerial vehicle deviates from the center of the picture according to the offset amount of the center of the target frame from the center of the image, and if the target unmanned aerial vehicle tracking is lost, controls the millimeter wave radar to re-perform detection and image recognition.

7. The neural network based real-time drone identification and tracking system of claim 1, wherein, The proportional control strategy is used for rapid response to the instantaneous offset of the target position in the image; the fuzzy control strategy is used for flexible adjustment when the offset amount is less than the preset threshold, and inhibits the false action caused by image jitter; the integral differential adjustment strategy is used for compensating the slow accumulation of small offset trend and accelerating the response in the case of severe offset.

8. The neural network-based real-time drone identification and tracking system of claim 1, wherein, After the tracking control module receives the identification frame coordinates, it calculates the fusion coefficient and control speed, the fused control speed instruction is packaged in a standardized JSON format, and is sent to the control service interface built based on the Flask framework through an HTTP POST request; after the control service interface receives the control instruction, it is translated into a continuous action command conforming to the ONVIF standard protocol, and real-time pan-tilt adjustment is realized through the built-in interface of the network camera; after the pan-tilt action is executed, the network camera continuously returns images, and the system performs a new round of identification, judgment and control instruction generation on each frame of image, forming a closed-loop tracking.

9. The neural network-based real-time drone identification and tracking system of claim 1, wherein, The tracking control module adopts a confidence degree sliding window fusion strategy in each control cycle: wherein, is the current confidence smoothing value, C i is the confidence of the i-th frame detection result, N is the window length, the confidence smoothing value is used to determine whether the target is continuously present or the confidence is continuously decreased, if it is considered that the target may be lost or the recognition is unstable, and a search or re-identification mode is entered; if it is considered that the target state is stable, and the current tracking strategy is maintained, wherein τ min , τ stable is a configurable threshold. 10.The real-time identification and tracking system for UAV based on neural network according to claim 1, wherein, The system adopts an RTSP+HTTP wireless communication mode, supports multi-thread asynchronous processing, multi-user concurrent access, and man-machine interaction of the visual control interface.

Images