An airborne supervisory device management method and system

Abstract

This invention relates to the field of equipment management technology and discloses a method and system for managing airborne monitoring equipment. The method involves: dividing the airborne monitoring equipment software into a hardware abstraction layer, a functional module layer, an application layer, and a communication layer; writing acquired image frames into a dedicated memory buffer; inputting the image frames from the dedicated memory buffer into a neural network processing unit for target detection to obtain obstacle recognition results and obstacle 3D distance data; inputting the image frames into a video processing unit for video encoding to obtain a compressed video stream; performing collision analysis based on the obstacle recognition results and obstacle 3D distance data; sending obstacle avoidance control commands to the flight control system; and encapsulating the compressed video stream via an uplink data stream before reporting it to the monitoring platform. This invention ensures the reliability and determinism of the emergency obstacle avoidance command execution path and improves the overall real-time response capability and system robustness of airborne monitoring equipment in multi-task concurrent scenarios.

Description

Technical Field

[0001] This invention relates to the field of equipment management technology, and in particular to a method and system for managing airborne monitoring equipment. Background Technology

[0002] Airborne surveillance equipment needs to simultaneously handle multiple heterogeneous computing tasks on a single hardware platform, including video acquisition, multi-sensor data fusion, target detection and inference, video encoding, and flight control command response. Existing airborne surveillance equipment uses a flat or shallow architecture for its embedded software, with tightly coupled functional modules. The path from video data acquisition to local storage must traverse the business application layer. If congestion occurs in the upper-layer business logic, local video storage is interrupted, and critical data cannot be preserved in fault scenarios. Furthermore, direct inter-module calls mean that any module's anomaly can easily propagate along the call chain to the entire system, resulting in severely insufficient system robustness.

[0003] Furthermore, while modern airborne processor systems-on-a-chip have integrated dedicated computing units such as neural network processing units and video processing units, existing software designs lack effective task mapping and scheduling mechanisms. Target detection inference and video encoding are still executed by the central processing unit in software, resulting in the central processing unit being under high load for a long time, the computing power of dedicated computing units being idle, and real-time control tasks not being able to obtain deterministic execution time guarantees. The end-to-end latency of obstacle avoidance response and emergency hovering commands is difficult to meet the stringent constraints of airborne safety applications. Summary of the Invention

[0004] The main objective of this invention is to provide a management method and system for airborne surveillance equipment. This invention solves the problem of loss of key video data in fault scenarios in the prior art, ensures the reliability and determinism of emergency obstacle avoidance command execution paths, and improves the overall real-time response capability and system robustness of airborne surveillance equipment in multi-task concurrent scenarios.

[0005] To achieve the above objectives, the present invention provides a method for managing airborne surveillance equipment, comprising the following steps: The software of the airborne surveillance equipment is divided into a hardware abstraction layer, a functional module layer, an application layer, and a communication layer, and the acquired image frames are written into a dedicated memory buffer. The image frames in the dedicated memory buffer are input into the neural network processing unit for target detection to obtain obstacle recognition results and obstacle 3D distance data. The image frames are then input into the video processing unit for video encoding to obtain a compressed video stream. Based on the obstacle recognition results and the obstacle's three-dimensional distance data, collision analysis is performed and obstacle avoidance control commands are sent to the flight control system. The compressed video stream is then encapsulated as an uplink data stream and reported to the monitoring platform.

[0006] Optionally, in a first implementation of the first aspect of the present invention, the step of dividing the airborne monitoring equipment software into a hardware abstraction layer, a functional module layer, an application layer, and a communication layer, and writing the acquired image frames into a dedicated memory buffer, includes: The airborne monitoring equipment software is divided into a hardware abstraction layer, a functional module layer, an application layer, and a communication layer. The hardware abstraction layer encapsulates camera acquisition operations with a video acquisition driver framework and inertial measurement unit sensor operations with a serial peripheral interface. The functional module layer is deployed independently in the form of data acquisition processes, data storage management processes, network transmission management processes, and algorithm management processes. The application layer runs the main control process using a first-in-first-out real-time scheduling strategy. The communication layer handles TCP / IP data transmission and reception with an independent network thread. Image frames captured by the binocular camera are written to a dedicated memory buffer.

[0007] Optionally, in a second implementation of the first aspect of the present invention, after writing the image frames captured by the binocular camera into a dedicated memory buffer, the method further includes: The dedicated memory buffer is managed by the data storage management process, and the image frames are written to the local solid storage medium, forming a local storage stream that bypasses the application layer. The sensor data frame stream is transferred to the main control process, encapsulated with a binary serialization protocol, and then reported through the communication layer to form an uplink data stream. The instructions issued by the regulatory platform are transmitted to the main control process through the communication layer. After the main control process completes the verification of the national cryptographic SM2 digital signature and cyclic redundancy check, the instructions are routed to the corresponding execution modules according to the instruction priority, forming a downlink data stream.

[0008] Optionally, in a third implementation of the first aspect of the present invention, the step of inputting the image frame in the dedicated memory buffer into the neural network processing unit for target detection to obtain obstacle recognition results and obstacle three-dimensional distance data, and inputting the image frame into the video processing unit for video encoding to obtain a compressed video stream, includes: The algorithm management process sends inference instructions to the neural network processing unit, inputs the image frames in the dedicated memory buffer into the neural network processing unit to perform target detection and binocular stereo matching in parallel, and obtains target detection box data and disparity map; the central processing unit performs multi-sensor fusion analysis in parallel during the inference process of the neural network processing unit to obtain pose fusion data; The target detection box data is input into the central processing unit and combined with the disparity map to generate obstacle recognition results and obstacle 3D distance data; The image frame is input into the video processing unit through the media processing platform interface to perform hardware encoding, resulting in a compressed video stream.

[0009] Optionally, in the fourth implementation of the first aspect of the present invention, the algorithm management process issues inference instructions to the neural network processing unit, inputs the image frames in the dedicated memory buffer into the neural network processing unit to perform target detection and binocular stereo matching in parallel, and obtains target detection box data and disparity map; the central processing unit performs multi-sensor fusion analysis in parallel during the inference process of the neural network processing unit to obtain pose fusion data, including: The image frames in the dedicated memory buffer are preprocessed to obtain preprocessed image frames, and the preprocessed image frames are mapped to the neural network processing unit. The algorithm management process sends inference instructions to the neural network processing unit. The neural network processing unit performs target detection according to the inference instruction and outputs target detection box data and confidence array; the image frame is input into the neural network processing unit to perform binocular stereo matching and calculate disparity map; after the neural network processing unit completes inference, it notifies the algorithm management process to obtain the inference result via hardware interrupt; During the inference process of the neural network processing unit, the central processing unit performs fusion analysis based on the latitude, longitude, velocity and heading data output by the satellite navigation and positioning module, the three-axis acceleration and angular velocity data output by the inertial measurement unit, and the altitude data output by the millimeter-wave radar to obtain attitude fusion data.

[0010] Optionally, in a fifth implementation of the first aspect of the present invention, the step of inputting the target detection box data into the central processing unit and combining it with the disparity map to generate obstacle recognition results and obstacle three-dimensional distance data includes: The target detection box data is input into the central processing unit to perform non-maximum suppression processing, select and retain the detection boxes, and then perform coordinate inverse normalization mapping on the retained detection boxes to obtain the set of obstacle pixel coordinates; Based on the set of obstacle pixel coordinates, query the disparity value of the corresponding pixel of each detected target in the disparity map, and calculate the three-dimensional distance data of the obstacle for each detected target by combining the focal length and binocular baseline distance in the camera calibration parameters. The obstacle pixel coordinate set is combined with the obstacle three-dimensional distance data to obtain the obstacle recognition result.

[0011] Optionally, in a sixth implementation of the first aspect of the present invention, the step of performing collision analysis based on the obstacle recognition result and the obstacle's three-dimensional distance data and sending obstacle avoidance control commands to the flight control system, and encapsulating the compressed video stream via an uplink data stream and reporting it to the monitoring platform, includes: Compare the three-dimensional distance data of the obstacle with a safe distance threshold; When the three-dimensional distance data of the obstacle is less than the safe distance threshold, a collision risk is determined and an obstacle avoidance control command is generated; The obstacle avoidance control command is sent to the flight control system via a serial communication interface. At the same time, the compressed video stream, the obstacle identification result, the obstacle three-dimensional distance data, and the alarm data of the obstacle avoidance action are encapsulated in binary serialization protocol via the uplink data stream and reported to the monitoring platform.

[0012] Optionally, in a seventh implementation of the first aspect of the present invention, the step of determining a collision risk and generating an obstacle avoidance control command when the three-dimensional distance data of the obstacle is less than the safe distance threshold includes: When the three-dimensional distance data of the obstacle is less than the safe distance threshold, a hovering command of the flight control communication protocol is generated to the flight control system, so that the UAV locks the current latitude, longitude and altitude. At the same time, based on the pixel position of each detected target in the obstacle recognition result and the three-dimensional distance data of the obstacle, the obstacle density in the current flight path is calculated to obtain the obstacle density statistics result and the path feasibility judgment result. When the path feasibility assessment result is that the path is feasible, path planning is performed based on the obstacle identification result, a detour path is generated and converted into an obstacle avoidance control command containing a waypoint instruction sequence; when the path feasibility assessment result is that the path is not feasible, an obstacle avoidance control command containing a return instruction is directly generated.

[0013] Optionally, in the eighth implementation of the first aspect of the present invention, the step of sending the obstacle avoidance control command to the flight control system via a serial communication interface, and simultaneously encapsulating the compressed video stream, the obstacle recognition result, the obstacle three-dimensional distance data, and the alarm data of the executed obstacle avoidance action via the uplink data stream using a binary serialization protocol and reporting them to the monitoring platform, includes: The thread scheduling strategy for processing the obstacle avoidance control command in the main control process is set to the highest priority. The obstacle avoidance control command is subjected to national cryptographic SM2 digital signature verification and cyclic redundancy check. After the verification is successful, it is sent to the flight control system through the serial communication interface. The compressed video stream and alarm data containing the obstacle identification results, the obstacle's three-dimensional distance data, and the obstacle avoidance actions performed are encapsulated into a reporting data packet using a binary serialization protocol, and then reported to the monitoring platform via the uplink data stream through the 4G link.

[0014] The present invention also provides an airborne surveillance equipment management device, comprising: The writing module is used to divide the airborne surveillance equipment software into a hardware abstraction layer, a functional module layer, an application layer, and a communication layer, and to write the acquired image frames into a dedicated memory buffer. The processing module is used to input the image frames in the dedicated memory buffer into the neural network processing unit for target detection, obtain obstacle recognition results and obstacle three-dimensional distance data, and input the image frames into the video processing unit for video encoding to obtain a compressed video stream; The control module is used to perform collision analysis based on the obstacle recognition results and the three-dimensional distance data of the obstacle, send obstacle avoidance control commands to the flight control system, and encapsulate the compressed video stream through an uplink data stream before reporting it to the monitoring platform.

[0015] In summary, this invention completely severs the random coupling between functional modules at the architectural level by dividing the airborne monitoring equipment software into a four-layer isolated architecture: a hardware abstraction layer, a functional module layer, an application layer, and a communication layer, and establishing three independent unidirectional data paths: a local storage stream, an uplink data stream, and a downlink data stream. The local storage stream bypasses the application layer, directly writing image frames to a dedicated memory buffer and then persisting them to disk. This completely isolates the video storage path from the upper-layer business logic in the process space. Any communication layer anomalies or business logic congestion do not affect the continuous storage of local video, solving the problem of critical video data loss in fault scenarios in existing technologies. By managing the process through algorithms to issue inference instructions to the neural network processing unit in a non-blocking manner, the target detection inference and binocular stereo matching tasks are offloaded to the neural network processing unit for execution, and the video encoding task is offloaded to the video processing unit for execution. This frees the central processing unit from the two heaviest computational burdens mentioned above, allowing it to focus on extended Kalman filtering multi-sensor fusion and real-time scheduling control, ensuring deterministic execution time guarantees for obstacle avoidance decision-making and flight control command response. Image frames are input to the neural network processing unit with zero copy via address registration, and video frames are registered to the video processing unit via the media processing platform interface with zero copy. This eliminates redundant memory copy overhead during data transfer between processing units, further reducing end-to-end processing latency. Obstacle avoidance control commands are sent with the highest priority via a first-in-first-out real-time scheduling strategy through the serial communication interface, coupled with a three-retransmission and takeover board backup transmission mechanism. This ensures the reliability and determinism of the emergency obstacle avoidance command execution path, improving the overall real-time response capability and system robustness of the airborne monitoring equipment in multi-task concurrent scenarios. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of an embodiment of the airborne surveillance equipment management method according to one embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the process of writing the acquired image frames into a dedicated memory buffer in an embodiment of the present invention; Figure 3 This is a schematic diagram of target detection and video encoding in an embodiment of the present invention; Figure 4This is a schematic diagram of collision analysis and obstacle avoidance control in an embodiment of the present invention; Figure 5 This is a structural block diagram of the airborne monitoring equipment management device in an embodiment of the present invention.

[0017] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0019] Reference Figure 1 This embodiment provides a method for managing airborne surveillance equipment, including the following steps: S100 divides the airborne surveillance equipment software into a hardware abstraction layer, a functional module layer, an application layer, and a communication layer, and writes the acquired image frames into a dedicated memory buffer. S200 inputs the image frames in the dedicated memory buffer into the neural network processing unit for target detection, obtains obstacle recognition results and obstacle 3D distance data, and inputs the image frames into the video processing unit for video encoding to obtain a compressed video stream; The S300 performs collision analysis based on obstacle recognition results and obstacle 3D distance data, sends obstacle avoidance control commands to the flight control system, and reports the compressed video stream to the monitoring platform after encapsulation via uplink data stream.

[0020] This invention manages the process to issue inference commands to the neural network processing unit in a non-blocking manner, offloading target detection inference and stereo matching tasks to the neural network processing unit, and video encoding tasks to the video processing unit. This frees the central processing unit from the two heaviest computational burdens, allowing it to focus on extended Kalman filtering, multi-sensor fusion, and real-time scheduling control, ensuring deterministic execution time for both obstacle avoidance decision-making and flight control command responses. Image frames are input to the neural network processing unit with zero-copy address registration, and video frames are registered to the video processing unit with zero-copy data via the media processing platform interface, eliminating redundant memory copy overhead during data transfer between processing units and further reducing end-to-end processing latency. Obstacle avoidance control commands are sent with the highest priority via a first-in-first-out real-time scheduling strategy through the serial communication interface, coupled with a three-retransmission and takeover board backup transmission mechanism, ensuring the reliability and determinism of emergency obstacle avoidance command execution paths. This improves the overall real-time response capability and system robustness of airborne surveillance equipment in multi-task concurrent scenarios.

[0021] In one example, such as Figure 2 The airborne surveillance equipment software is divided into a hardware abstraction layer, a functional module layer, an application layer, and a communication layer. The acquired image frames are written to a dedicated memory buffer, including: S101 divides the airborne surveillance equipment software into a hardware abstraction layer, a functional module layer, an application layer, and a communication layer. The hardware abstraction layer encapsulates camera acquisition operations with a video acquisition driver framework and inertial measurement unit sensor operations with a serial peripheral interface. The functional module layer is deployed independently in the form of data acquisition processes, data storage management processes, network transmission management processes, and algorithm management processes. The application layer runs the main control process with a first-in-first-out real-time scheduling strategy. The communication layer handles TCP / IP data transmission and reception with an independent network thread. S102 writes the image frames captured by the binocular camera into a dedicated memory buffer.

[0022] In this example, the software architecture of the airborne surveillance equipment is constructed into a four-layer operating structure, with image acquisition, sensor access, service scheduling, and network communication running in different software units. This reduces cross-module coupling at the architectural level and reserves stable data entry points for target detection, video encoding, obstacle avoidance control, and link reporting. The hardware abstraction layer is located at the bottom and undertakes the task of unifying access to the underlying hardware. The camera part uses a video acquisition driver framework to encapsulate the initialization, frame synchronization, buffer allocation, acquisition startup, and frame extraction processes of the binocular camera, so that the upper layer only needs to call the unified acquisition interface to obtain the left and right eye image frames corresponding to the time. The inertial measurement unit uses a serial peripheral interface to complete register reading and writing, status polling, sampling triggering, and raw data reading, and organizes the results such as acceleration and angular velocity into a standard data structure to provide upwards, thereby avoiding the application layer directly accessing the underlying register addresses and timing details. The functional module layer is located above the hardware abstraction layer and is divided into data acquisition processes, data storage management processes, network transmission management processes, and algorithm management processes. Each process focuses on data acquisition and processing, local disk storage management, link status maintenance, and heterogeneous computing scheduling, respectively. Processes coordinate with each other through message queues, shared memory, or event notifications, which maintains operational isolation and facilitates control over the execution boundaries of different tasks in high-concurrency scenarios. The application layer uses the master control process as the core of business organization and adopts a first-in-first-out real-time scheduling strategy to maintain the order of request processing. This allows the master control process to receive image acquisition results, sensor status, and network events according to a predetermined time sequence, and integrates uplink business encapsulation, downlink command distribution, and obstacle avoidance logic into a unified control entry point. The communication layer maintains an independent network thread structure, dedicated to TCP / IP data transmission and reception, ensuring that network congestion does not directly intrude into the image acquisition and storage path.

[0023] The system synchronously acquires left and right eye images using a binocular camera under the control of a video acquisition driver framework. The driver side writes the acquired image frames directly into a dedicated memory buffer according to a predetermined frame format, timestamp rules, and buffer address mapping. This dedicated memory buffer is pre-allocated and its addresses are registered during system startup, allowing controlled access to the same buffer resource by the acquisition, storage, and algorithm sides. To reduce overhead during inter-layer image frame transfer, the write path uses memory mapping or address registration. This allows the raw frames acquired by the driver to be directly read by the data storage management process after entering the dedicated memory buffer, and also to be extracted by the algorithm management process according to access permissions for inference preparation, maintaining the short-link characteristic of the image data path. Simultaneously, when binocular image frames are written to the dedicated memory buffer, management information such as frame sequence number, acquisition time, left and right target identification, and buffer occupancy status needs to be established. This enables the data storage management process to perform circular coverage control, abnormal frame removal, and continuous frame persistence based on these identifiers.

[0024] In one example, after writing the image frames captured by the binoculars to a dedicated memory buffer, the process also includes: The data storage management process manages a dedicated memory buffer and writes image frames to local solid storage media, forming a local storage stream that bypasses the application layer. The sensor data frame stream is transferred to the main control process, encapsulated with a binary serialization protocol, and then reported through the communication layer to form an uplink data stream; The instructions issued by the regulatory platform are transmitted to the main control process through the communication layer. After the main control process completes the verification of the national cryptographic SM2 digital signature and cyclic redundancy check, the instructions are routed to the corresponding execution modules according to the instruction priority, forming a downlink data stream.

[0025] In this example, the data storage management process continuously takes over the read and write management functions of the dedicated memory buffer. Within the buffer, it establishes a cache management relationship organized by frame sequence number, timestamp, and read / write pointers, and uses a circular buffer mechanism to maintain continuous image frame writing. After the binocular camera completes acquisition, the image frames are first written to the dedicated memory buffer by the underlying acquisition interface. Then, the data storage management process directly extracts the frames to be written from the dedicated memory buffer and places them into the local storage medium. This forms a direct write link from the hardware abstraction layer to the functional module layer and then to the local storage medium. The entire disk writing process no longer passes through the application layer where the main control process resides, nor does it rely on the communication layer for forwarding. Structurally, it cuts off the interference of business logic congestion on the video storage path, ensuring that communication thread blocking, protocol parsing delays, or upper-layer task congestion will not affect continuous video storage. To maintain a non-blocking operation state for the local storage stream, the data storage management process sets up an independent cache domain, independent wake-up conditions, and an independent disk writing thread for the dedicated memory buffer. When an alarm, abnormal event, or manual extraction is detected, the image frames within the corresponding time window are locked, prioritized for writing, and persistently marked, thus ensuring that critical flight footage is completely preserved.

[0026] The uplink data stream is organized by the data acquisition process, which organizes multi-source data from the inertial measurement unit, positioning module, and other sensors into a unified data frame and transfers it to the main control process. The main control process calls the binary serialization protocol to complete the encapsulation according to the predetermined message structure, and then submits the encapsulation result to the communication layer. An independent network thread executes TCP / IP transmission, thus forming a sequential reporting path from the functional module layer to the application layer and then to the communication layer. With the adoption of the binary serialization protocol, the message length is easier to control, the field parsing process is more direct, and the communication layer is only responsible for sending and receiving without undertaking business judgment tasks. This helps to reduce the software burden outside the main control process and compress the reporting latency.

[0027] Downlink data flow is opposite to uplink data flow. Control commands issued by the regulatory platform first enter the communication layer receiving thread. After the communication layer completes the reception, it delivers the command data to the main control process. The main control process first performs SM2 digital signature verification, and then performs cyclic redundancy check. Only when the identity authenticity and message integrity requirements are met will it continue to the command distribution process. Subsequently, the main control process establishes a priority routing relationship based on the command type, risk level, and execution urgency. High-priority control commands are sent to the corresponding execution modules first, while ordinary configuration commands are sent to the network transmission management process, data storage management process, or other control units in the usual order. This forms a downlink control path from the communication layer to the application layer, then to the functional module layer, and finally to the underlying execution interface.

[0028] In one example, such as Figure 3 The image frames in the dedicated memory buffer are input into the neural network processing unit for target detection to obtain obstacle recognition results and obstacle 3D distance data. The image frames are then input into the video processing unit for video encoding to obtain a compressed video stream, including: S201, the algorithm management process sends inference instructions to the neural network processing unit, inputs the image frames in the dedicated memory buffer into the neural network processing unit to perform target detection and binocular stereo matching in parallel, and obtains target detection box data and disparity map. During the inference process of the neural network processing unit, the central processing unit performs multi-sensor fusion analysis in parallel to obtain pose fusion data. S202, Input the target detection box data into the central processing unit and combine it with the disparity map to generate obstacle recognition results and obstacle 3D distance data; S203 inputs the image frame into the video processing unit through the media processing platform interface to perform hardware encoding and obtain a compressed video stream.

[0029] In this example, the algorithm management process reads the corresponding binocular image frame from a dedicated memory buffer and performs preprocessing operations such as input size adjustment, format conversion, and mapping registration on the central processing unit (CPU). The preprocessed image frame is then sent to the neural network processing unit (NN unit), which simultaneously issues inference instructions to the NN unit. This enables the NN unit to perform target detection and binocular stereo matching in parallel around the same batch of input data, thereby outputting target detection box data, disparity map, and attitude fusion data corresponding to the current flight state. While the NN unit is in inference mode, the CPU synchronously receives data from the satellite navigation and positioning module, inertial measurement unit, and millimeter-wave radar and performs fusion analysis to ensure that the visual results and flight state results are aligned in time. After the NN unit completes inference, a hardware interrupt is used to notify the algorithm management process to extract the inference output results.

[0030] The central processing unit (CPU) receives target detection bounding box data, filters and restores the coordinates of the detection results, and then combines the depth information corresponding to the detected target in the disparity map to generate obstacle recognition results. Simultaneously, it calculates the 3D distance data of the obstacle based on the calibration parameters required for binocular ranging, ensuring that the output results simultaneously provide a basis for determining the target category, target location, and spatial distance. After completing the visual analysis path, a video backhaul path is constructed using the same batch of image frames. The application side or video processing thread registers the image frame addresses with the video processing unit through the media processing platform interface. The video processing unit directly executes hardware encoding to form a compressed video stream. The CPU is only responsible for configuring input parameters, managing encoding sessions, and reading encoding results, without bearing the actual encoding computational load, thus avoiding heavy resource contention between target detection, stereo matching, and video compression.

[0031] In one example, the algorithm management process issues inference instructions to the neural network processing unit, inputting image frames from a dedicated memory buffer into the neural network processing unit to perform object detection and stereo matching in parallel, obtaining object detection bounding box data and disparity maps; the central processing unit performs multi-sensor fusion analysis in parallel during the inference process of the neural network processing unit, obtaining pose fusion data, including: The image frames in the dedicated memory buffer are preprocessed to obtain preprocessed image frames, and the preprocessed image frames are mapped to the neural network processing unit. The algorithm management process then issues inference instructions to the neural network processing unit. The neural network processing unit performs target detection according to the inference instructions and outputs target detection box data and confidence array; it inputs image frames into the neural network processing unit to perform binocular stereo matching and calculate disparity map; after the neural network processing unit completes inference, it notifies the algorithm management process to obtain the inference result via hardware interrupt. During the inference process of the neural network processing unit, the central processing unit performs fusion analysis based on the latitude, longitude, velocity and heading data output by the satellite navigation and positioning module, the three-axis acceleration and angular velocity data output by the inertial measurement unit, and the altitude data output by the millimeter-wave radar to obtain attitude fusion data.

[0032] In this example, the algorithm management process reads left and right eye image data from a dedicated memory buffer based on the address information of the current acquisition frame. On the central processing unit (CPU) side, it performs preparatory operations such as size normalization, pixel format conversion, and numerical normalization to ensure that the image data entering the inference chain meets the input conditions of the neural network processing unit in terms of resolution, channel arrangement, and data type. The preprocessed image frames are registered to the input area accessible to the neural network processing unit via address registration or memory mapping, minimizing redundant copying during the transfer of image data from the dedicated memory buffer to the neural network processing unit. Simultaneously, the algorithm management process issues corresponding inference instructions based on the current task type, task priority, and the idle state of the neural network processing unit, establishing a stable task initiation entry point. Upon receiving the inference instructions, the neural network processing unit performs object detection calculations on the input image, outputting object detection bounding box data and a confidence array. Simultaneously, it performs stereo matching calculations on the binocular image pairs, generating a disparity map corresponding to the scene's spatial depth distribution. These two computational chains operate in parallel around the same batch of image data. After object detection and stereo matching are completed, the neural network processing unit sends an inference termination notification to the algorithm management process via a hardware interrupt. The algorithm management process then enters the result extraction stage, reads the object detection box data, confidence array and disparity map results according to the predetermined output interface, and writes these results into the result buffer that can be directly accessed by the central processing unit.

[0033] Simultaneously, within the entire time window of the neural network processing unit's inference execution, the central processing unit (CPU) receives latitude, longitude, velocity, and heading data output by the satellite navigation and positioning module, triaxial acceleration and angular velocity data output by the inertial measurement unit, and altitude data output by the millimeter-wave radar. It aligns, denoises, and fuses the multi-source data according to a unified time reference, forming attitude fusion data that matches the current visual frame. The visual inference link and the flight state perception link proceed synchronously in time. The algorithm management process is responsible for initiating tasks and retrieving results from the neural network processing unit, while the CPU is responsible for the parallel fusion of multi-source state variables. This allows for the simultaneous acquisition of target detection box data, confidence arrays, disparity maps, and attitude fusion data reflecting flight attitude and motion state.

[0034] In one example, object detection bounding box data is input into the central processing unit and combined with a disparity map to generate obstacle recognition results and 3D distance data of the obstacles, including: The target detection box data is input into the central processing unit to perform non-maximum suppression processing, the retained detection boxes are selected, and then the coordinate inverse normalization mapping is performed on the retained detection boxes to obtain the set of obstacle pixel coordinates; Based on the set of obstacle pixel coordinates, query the disparity value of the corresponding pixel point of each detected target in the disparity map, and calculate the obstacle three-dimensional distance data of each detected target by combining the focal length and binocular baseline distance in the camera calibration parameters. The obstacle pixel coordinate set is merged with the obstacle 3D distance data to obtain the obstacle recognition result.

[0035] In this example, after receiving the target detection bounding box data and disparity map returned by the neural network processing unit, the central processing unit (CPU) sends overlapping candidate boxes corresponding to the same target to the non-maximum suppression (NMS) processing flow. Redundant candidate boxes are progressively eliminated based on confidence level and overlap relationship, retaining only the detection boxes that represent the actual target boundaries. This compresses duplicate recognition results and stabilizes the distance matching basis. The CPU performs coordinate denormalization mapping on the retained detection boxes, restoring the boundary positions in the model input coordinate domain to the original image coordinate domain, forming a set of obstacle pixel coordinates that directly correspond to image pixel positions. The CPU then queries the disparity value of the corresponding pixel in the disparity map based on the obstacle pixel coordinate set, and inputs the obtained disparity value, along with the focal length and binocular baseline distance from the camera calibration parameters, into the binocular ranging calculation process. Complete the acquisition of obstacle 3D distance data for each detection target, among which... This represents the three-dimensional distance data of the obstacle, in meters. The camera focal length is expressed in pixels. This indicates the binocular baseline distance, in meters. The parallax value is expressed in pixels. Since the focal length and parallax value use the same pixel scale, and the binocular baseline distance uses the metric unit, the resulting distance can directly characterize the spatial distance between the target and the airborne surveillance equipment.

[0036] The obstacle pixel coordinate set and obstacle 3D distance data are merged at the target level. A unified result recording unit is established for each detected target, so that the same result recording unit contains target category, image position and spatial distance information, and generates obstacle recognition results.

[0037] In one example, such as Figure 4 Based on obstacle recognition results and obstacle 3D distance data, collision analysis is performed and obstacle avoidance control commands are sent to the flight control system. The compressed video stream is encapsulated via uplink data stream and then reported to the monitoring platform, including: S301, compare the three-dimensional distance data of the obstacle with the safe distance threshold; S302, when the three-dimensional distance data of the obstacle is less than the safe distance threshold, it is determined that there is a collision risk and an obstacle avoidance control command is generated; S303 sends obstacle avoidance control commands to the flight control system via a serial communication interface. At the same time, it encapsulates the compressed video stream, obstacle recognition results, obstacle 3D distance data, and alarm data of executed obstacle avoidance actions into an uplink data stream using a binary serialization protocol and reports it to the monitoring platform.

[0038] In this example, after receiving the structured obstacle list, the main control process loads the 3D distance data of each detected target into the safe distance threshold determination module in the running memory, and performs distance comparison item by item in a target-level manner. When the obstacle's 3D distance data is greater than or equal to the safe distance threshold, the main control process maintains the current flight mission state and only continues to perform routine status monitoring and link reporting, thereby ensuring the stable operation of the perception and communication links in ordinary scenarios. When the obstacle's 3D distance data is less than the safe distance threshold, the main control process determines that the current flight path has entered the collision risk zone and elevates the risk target from a general perception result to a high-priority object that needs to enter the control and handling process. The main control process forms a risk profile of the current flight path based on the target position distribution and obstacle 3D distance data in the obstacle recognition results. When necessary, it first generates hovering obstacle avoidance control commands to make the UAV immediately enter the current position holding state. On the basis of hovering stability, it continues to combine the obstacle density and path feasibility judgment results within the flight path to determine the subsequent handling direction. If the path is feasible, it switches to detour control generation; if the path is infeasible, it switches to return-to-home control generation.

[0039] The control execution link and the regulatory reporting link are activated simultaneously. One link connects to the flight control system to complete the real-time control closed loop, while the other connects to the regulatory platform to complete the status alarm closed loop. To ensure control timeliness, the thread scheduling strategy for processing obstacle avoidance control commands in the main control process is elevated to the highest priority, giving obstacle avoidance control commands priority execution at the central processing unit scheduling level. Subsequently, the obstacle avoidance control commands undergo a security verification process, including verification of the national cryptographic SM2 digital signature and cyclic redundancy check. After both the authenticity of the identity and the integrity of the message are satisfied, they are sent to the flight control system via the serial communication interface, enabling the flight control system to receive hovering, detour, or return-to-home control commands in a timely manner. At the same time, the compressed video stream, obstacle recognition results, obstacle 3D distance data, and alarm data of executed obstacle avoidance actions are all sent to the uplink encapsulation link. The main control process generates a unified reporting data packet according to the binary serialization protocol, which is then sent to the communication layer via the uplink data stream and transmitted to the regulatory platform through the 4G link, enabling the regulatory platform to simultaneously obtain visual images, risk target information, spatial distance information, and control execution information.

[0040] In one example, when the 3D distance data of an obstacle is less than a safe distance threshold, a collision risk is identified and obstacle avoidance control instructions are generated, including: When the three-dimensional distance data of the obstacle is less than the safe distance threshold, a hovering command is generated to the flight control system via the flight control communication protocol, so that the UAV locks the current latitude, longitude and altitude. At the same time, based on the pixel position of each detected target and the three-dimensional distance data of the obstacle in the obstacle recognition result, the obstacle density in the current flight path is calculated to obtain the obstacle density statistics result and the path feasibility judgment result. When the path feasibility assessment result is that the path is feasible, path planning is performed based on the obstacle identification result, a detour path is generated and converted into an obstacle avoidance control command containing a waypoint instruction sequence; when the path feasibility assessment result is that the path is not feasible, an obstacle avoidance control command containing a return instruction is directly generated.

[0041] In this example, after receiving the collision risk assessment result, the master control process raises the current risk target to the highest response level and first generates a hovering command in the flight control communication protocol, so that the flight control system enters the position holding state first. The flight control system calls onboard positioning, barometric altimeter and inertial measurement information to lock the UAV at the current latitude, longitude and altitude corresponding to the risk triggering time, so as to obtain a stable decision time window for path replanning. Among them, the safe distance threshold is pre-written into the local parameter file as an operation judgment parameter and loaded into the running memory when the device starts up. It can also be remotely updated through parameter configuration commands during the flight mission. Simultaneously with the hovering command being issued, the main control process reads the pixel positions of each detected target from the obstacle recognition results. Combining this with the corresponding 3D distance data of the obstacles, it converts each detected target into obstacle spatial coordinates in the aircraft coordinate system. Then, it performs spatial statistics on the obstacle distribution along the current flight path, dividing the forward safety corridor, lateral buffer zone, and candidate detour area into a unified analysis window. Based on the obstacle space occupancy density, target spacing, near-field target clustering, and continuous passable width within each area, it generates obstacle density statistics and outputs a path feasibility assessment result. If the path feasibility assessment result indicates that the path is feasible, the path planning module plans the path around the current hovering position and the target... The system reconstructs local passageways based on the direction of travel and the spatial distribution of obstacles. It generates detour paths while avoiding high-density obstacle clusters, and discretizes these paths into a sequence of waypoint commands containing multiple waypoint coordinates, heading adjustments, and stage speed constraints. The master control process then encapsulates this sequence into obstacle avoidance control commands, allowing the flight control system to perform yaw, side-shift, and return-to-home actions segment by segment according to a predetermined waypoint sequence. If the path feasibility assessment indicates that the path is infeasible, the master control process stops resolving local detours and directly generates obstacle avoidance control commands containing return-to-home instructions. This causes the flight control system to exit the current mission path and switch to return-to-home control, returning to the takeoff point or a pre-defined safe area along a preset safe path. Hovering, path evaluation, and obstacle avoidance control generation are continuously scheduled. First, the UAV's current position is fixed. Then, obstacle density statistics and path feasibility assessments are completed. Subsequently, based on the assessment results, a one-way selection is made between detour control and return-to-home control, ensuring that the control link always revolves around the same set of obstacle identification results and spatial distance data at the same time.

[0042] In one example, obstacle avoidance control commands are sent to the flight control system via a serial communication interface. Simultaneously, compressed video streams, obstacle recognition results, obstacle 3D distance data, and alarm data from executed obstacle avoidance actions are encapsulated in a binary serialization protocol via an uplink data stream and reported to the monitoring platform. This includes: Set the thread scheduling strategy for processing obstacle avoidance control commands in the main control process to the highest priority, perform national cryptographic SM2 digital signature verification and cyclic redundancy check on the obstacle avoidance control commands, and send them to the flight control system through the serial communication interface after successful verification. The compressed video stream, along with alarm data containing obstacle recognition results, obstacle 3D distance data, and obstacle avoidance actions, is encapsulated into a reporting data packet using a binary serialization protocol and then reported to the monitoring platform via the uplink data stream and 4G link.

[0043] In this example, a dedicated processing thread is set up within the main control process for obstacle avoidance control commands, and the scheduling strategy of the dedicated processing thread is elevated to the highest priority. This allows the obstacle avoidance control commands to be executed on the central processing unit side before routine status reporting, log organization, and non-critical business tasks, thereby reducing the scheduling waiting time between the completion of collision risk assessment and the actual receipt of control commands by the flight control system. Before the obstacle avoidance control command enters the transmission link, the master control process calls the device-side cryptographic processing module to perform SM2 digital signature verification on the obstacle avoidance control command. This verifies the legitimacy of the command source and the absence of forgery of the command content. After the SM2 digital signature verification is successful, cyclic redundancy check is performed to confirm that no bit-level damage has occurred during the reception, buffering, and internal circulation of the command message. Only when both the authenticity of the identity and the integrity of the message are met can the master control process allow the obstacle avoidance control command to enter the serial transmission stage. After completing the dual verification, the master control process converts the hovering command, detour command, or return-to-home command into a flight control communication protocol command frame that the flight control system can directly parse, and sends it to the flight control system via the serial communication interface. For example, the UART physical channel can be used to complete the control message transmission between the airborne monitoring equipment and the flight control system. The emergency hovering command processing thread is elevated to the highest priority of SCHED_FIFO, and sent via the UART interface at 115200bps. In case of transmission failure, a real-time execution mode of retransmission and backup transmission switching is performed, thereby enabling the control link to maintain strong determinism and reliability in high-concurrency scenarios.

[0044] After packaging video evidence, perception results, and control results, they are sent to the uplink data stream for further transmission. The compressed video stream output by the video processing unit is read by the main control process and aggregated with obstacle recognition results, obstacle 3D distance data, and alarm data corresponding to the executed obstacle avoidance actions into a unified business data set. The main control process then calls the binary serialization protocol to complete field-level encapsulation, forming a compact, controllable-length, and easily parsable uplink data packet. The uplink data packet enters the communication layer along the predetermined uplink data stream and is sent to the monitoring platform via the 4G link. This allows the monitoring platform to simultaneously acquire the compressed video footage, target category, target pixel position, spatial distance, and records of triggered hovering, detour, or return actions, forming a closed-loop reporting structure that corresponds consistently between perception results, control actions, and monitoring traces. For example, binary Protocol Buffers can be used instead of text format between the application layer and the platform to reduce message size and parsing overhead. The video stream can enter the uplink encapsulation path after the VPU completes H.264 or H.265 hardware encoding, and then be uploaded in real time via the 4G network. This reduces the burden on the central processing unit in the encoding and communication process and maintains the real-time response capability of the airborne monitoring equipment in a multi-task concurrent environment.

[0045] Reference Figure 5 This embodiment provides an airborne surveillance equipment management device 500, including: The writing module 501 is used to divide the airborne monitoring equipment software into a hardware abstraction layer, a functional module layer, an application layer, and a communication layer, and to write the acquired image frames into a dedicated memory buffer. Processing module 502 is used to input image frames in a dedicated memory buffer into a neural network processing unit for target detection, obtain obstacle recognition results and obstacle 3D distance data, and input image frames into a video processing unit for video encoding to obtain a compressed video stream; The control module 503 is used to perform collision analysis based on obstacle recognition results and obstacle 3D distance data, send obstacle avoidance control commands to the flight control system, and encapsulate the compressed video stream through uplink data stream before reporting it to the monitoring platform.

[0046] In this embodiment, the specific implementation of each unit in the above device embodiment is described in the above method embodiment, and will not be repeated here.

[0047] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, apparatus, article, or method. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, apparatus, article, or method that includes that element.

[0048] The above description is only a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. An airborne supervisory device management method characterized by comprising: include: The software of the airborne surveillance equipment is divided into a hardware abstraction layer, a functional module layer, an application layer, and a communication layer, and the acquired image frames are written into a dedicated memory buffer. The image frames in the dedicated memory buffer are input into the neural network processing unit for target detection to obtain obstacle recognition results and obstacle 3D distance data. The image frames are then input into the video processing unit for video encoding to obtain a compressed video stream. Based on the obstacle recognition results and the obstacle's three-dimensional distance data, collision analysis is performed and obstacle avoidance control commands are sent to the flight control system. The compressed video stream is then encapsulated as an uplink data stream and reported to the monitoring platform.

2. The on-board supervisory device management method according to claim 1, characterized by, The process of dividing the airborne surveillance equipment software into a hardware abstraction layer, a functional module layer, an application layer, and a communication layer, and writing the acquired image frames into a dedicated memory buffer, includes: The airborne monitoring equipment software is divided into a hardware abstraction layer, a functional module layer, an application layer, and a communication layer. The hardware abstraction layer encapsulates camera acquisition operations with a video acquisition driver framework and inertial measurement unit sensor operations with a serial peripheral interface. The functional module layer is deployed independently in the form of data acquisition processes, data storage management processes, network transmission management processes, and algorithm management processes. The application layer runs the main control process using a first-in-first-out real-time scheduling strategy. The communication layer handles TCP / IP data transmission and reception with an independent network thread. Image frames captured by the binocular camera are written to a dedicated memory buffer.

3. The airborne surveillance equipment management method according to claim 2, characterized in that, After writing the image frames captured by the binocular cameras into a dedicated memory buffer, the process also includes: The dedicated memory buffer is managed by the data storage management process, and the image frames are written to the local solid storage medium, forming a local storage stream that bypasses the application layer. The sensor data frame stream is transferred to the main control process, encapsulated with a binary serialization protocol, and then reported through the communication layer to form an uplink data stream. The instructions issued by the regulatory platform are transmitted to the main control process through the communication layer. After the main control process completes the verification of the national cryptographic SM2 digital signature and cyclic redundancy check, the instructions are routed to the corresponding execution modules according to the instruction priority, forming a downlink data stream.

4. The airborne surveillance equipment management method according to claim 3, characterized in that, The process of inputting image frames from the dedicated memory buffer into a neural network processing unit for target detection to obtain obstacle recognition results and obstacle 3D distance data, and then inputting the image frames into a video processing unit for video encoding to obtain a compressed video stream, includes: The algorithm management process sends inference instructions to the neural network processing unit, inputs the image frames in the dedicated memory buffer into the neural network processing unit to perform target detection and binocular stereo matching in parallel, and obtains target detection box data and disparity map; the central processing unit performs multi-sensor fusion analysis in parallel during the inference process of the neural network processing unit to obtain pose fusion data; The target detection box data is input into the central processing unit and combined with the disparity map to generate obstacle recognition results and obstacle 3D distance data; The image frame is input into the video processing unit through the media processing platform interface to perform hardware encoding, resulting in a compressed video stream.

5. The airborne surveillance equipment management method according to claim 4, characterized in that, The algorithm management process sends inference instructions to the neural network processing unit, and inputs the image frames in the dedicated memory buffer into the neural network processing unit to perform target detection and binocular stereo matching in parallel, thereby obtaining target detection box data and disparity map; The central processing unit performs multi-sensor fusion analysis in parallel during inference by the neural network processing unit to obtain pose fusion data, including: The image frames in the dedicated memory buffer are preprocessed to obtain preprocessed image frames, and the preprocessed image frames are mapped to the neural network processing unit. The algorithm management process sends inference instructions to the neural network processing unit. The neural network processing unit performs target detection according to the inference instructions and outputs target detection box data and a confidence array; the image frame is input into the neural network processing unit to perform binocular stereo matching and calculate the disparity map; after the neural network processing unit completes the inference, it notifies the algorithm management process to obtain the inference result via a hardware interrupt. During the inference process of the neural network processing unit, the central processing unit performs fusion analysis based on the latitude, longitude, velocity and heading data output by the satellite navigation and positioning module, the three-axis acceleration and angular velocity data output by the inertial measurement unit, and the altitude data output by the millimeter-wave radar to obtain attitude fusion data.

6. The airborne surveillance equipment management method according to claim 4, characterized in that, The step of inputting the target detection box data into the central processing unit and combining it with the disparity map to generate obstacle recognition results and obstacle 3D distance data includes: The target detection box data is input into the central processing unit to perform non-maximum suppression processing, select and retain the detection boxes, and then perform coordinate inverse normalization mapping on the retained detection boxes to obtain the set of obstacle pixel coordinates; Based on the set of obstacle pixel coordinates, query the disparity value of the corresponding pixel of each detected target in the disparity map, and calculate the three-dimensional distance data of the obstacle for each detected target by combining the focal length and binocular baseline distance in the camera calibration parameters. The obstacle pixel coordinate set is combined with the obstacle three-dimensional distance data to obtain the obstacle recognition result.

7. The airborne surveillance equipment management method according to claim 6, characterized in that, The process of performing collision analysis based on the obstacle recognition results and the obstacle's three-dimensional distance data, sending obstacle avoidance control commands to the flight control system, and uploading the compressed video stream to the monitoring platform after encapsulation via uplink data stream includes: Compare the three-dimensional distance data of the obstacle with a safe distance threshold; When the three-dimensional distance data of the obstacle is less than the safe distance threshold, a collision risk is determined and an obstacle avoidance control command is generated; The obstacle avoidance control command is sent to the flight control system via a serial communication interface. At the same time, the compressed video stream, the obstacle identification result, the obstacle three-dimensional distance data, and the alarm data of the obstacle avoidance action are encapsulated in binary serialization protocol via the uplink data stream and reported to the monitoring platform.

8. The airborne surveillance equipment management method according to claim 7, characterized in that, When the three-dimensional distance data of the obstacle is less than the safe distance threshold, a collision risk is determined and an obstacle avoidance control command is generated, including: When the three-dimensional distance data of the obstacle is less than the safe distance threshold, a hovering command of the flight control communication protocol is generated to the flight control system, so that the UAV locks the current latitude, longitude and altitude. At the same time, based on the pixel position of each detected target in the obstacle recognition result and the three-dimensional distance data of the obstacle, the obstacle density in the current flight path is calculated to obtain the obstacle density statistics result and the path feasibility judgment result. When the path feasibility assessment result is that the path is feasible, path planning is performed based on the obstacle identification result, a detour path is generated and converted into an obstacle avoidance control command containing a waypoint instruction sequence; when the path feasibility assessment result is that the path is not feasible, an obstacle avoidance control command containing a return instruction is directly generated.

9. The airborne surveillance equipment management method according to claim 7, characterized in that, The process of sending the obstacle avoidance control command to the flight control system via a serial communication interface, and simultaneously encapsulating the compressed video stream, obstacle recognition results, obstacle 3D distance data, and alarm data of executed obstacle avoidance actions via the uplink data stream using a binary serialization protocol, and reporting them to the monitoring platform includes: The thread scheduling strategy for processing the obstacle avoidance control command in the main control process is set to the highest priority. The obstacle avoidance control command is subjected to national cryptographic SM2 digital signature verification and cyclic redundancy check. After the verification is successful, it is sent to the flight control system through the serial communication interface. The compressed video stream and alarm data containing the obstacle identification results, the obstacle's three-dimensional distance data, and the obstacle avoidance actions performed are encapsulated into a reporting data packet using a binary serialization protocol, and then reported to the monitoring platform via the uplink data stream through the 4G link.

10. A management device for airborne surveillance equipment, characterized in that, The steps for implementing the airborne surveillance equipment management method according to any one of claims 1 to 9 include: The writing module is used to divide the airborne surveillance equipment software into a hardware abstraction layer, a functional module layer, an application layer, and a communication layer, and to write the acquired image frames into a dedicated memory buffer. The processing module is used to input the image frames in the dedicated memory buffer into the neural network processing unit for target detection, obtain obstacle recognition results and obstacle three-dimensional distance data, and input the image frames into the video processing unit for video encoding to obtain a compressed video stream; The control module is used to perform collision analysis based on the obstacle recognition results and the three-dimensional distance data of the obstacle, send obstacle avoidance control commands to the flight control system, and encapsulate the compressed video stream through an uplink data stream before reporting it to the monitoring platform.

Images