Panoramic aircraft monitoring system and method based on video stitching and electronic map

Abstract

The invention belongs to the technical field of civil transportation aviation airport aircraft monitoring, and discloses a panoramic aircraft monitoring system and method based on video stitching and an electronic map. The method comprises the following steps: acquiring a vehicle position, a personnel position, aircraft information and flight dynamic information, and acquiring high-definition camera video stream data by a video gateway; performing video stitching processing on the acquired high-definition camera video stream data; the electronic map GIS server provides an electronic map service; carrying out panoramic video fusion processing on the electronic map and the spliced video, calculating the position information of the aircraft, and completing personnel management, vehicle management, flight dynamic information management and monitoring mode management; and reporting normal monitoring and exception handling conditions to the server through the client. According to the invention, a video fusion and electronic map function complementarity technology is adopted, all-weather panoramic video safety monitoring of the aircraft in different modes is realized, and invading personnel and vehicles can be actively found and alarmed.

Description

Technical Field

[0001] The present invention belongs to the technical field of aircraft monitoring at civil transport airports, and in particular relates to a panoramic aircraft monitoring system and method based on video splicing and electronic maps. Background Art

[0002] With the rapid development of the aviation industry, aircraft monitoring has become a key issue in air defense security. In the face of the severe air defense security situation, how to use technology to reduce the workload of aircraft monitors, improve monitoring capabilities, and ensure aircraft safety is an issue that must be studied and addressed.

[0003] According to statistics, the majority of civil aviation accidents occur on the airport ground. Operational safety during airport parking is a crucial component of civil aviation safety. There are generally two approaches to monitoring aircraft parked for long periods of time, aircraft not carrying out flight operations, or aircraft temporarily stopping on the apron during flight operations. The first is a regional monitoring approach, which requires high-level airport security facilities and management. The second is the existing monitoring model, which assigns monitoring staff based on the number of seats on an aircraft. Due to the technical and management limitations of the first approach, the vast majority of domestic airports still use the second approach for aircraft monitoring. The second approach, a product of years of security experience, has proven to be highly effective in practice. However, this model inefficiently utilizes human resources, fails to fully leverage the advancements in physical and technical security, and is inconsistent with current airport security technology. It is therefore difficult to adapt to the rapid development of the air transport industry and the demands of innovation in aviation security management systems.

[0004] In civil aviation, the most common technical approach is to monitor aircraft at airports through video surveillance systems. At some airports, intelligent analysis of video streams has enabled automated identification of common, primary events, such as the start and end times of vehicle support and aircraft entry and exit times. However, due to imperfect surveillance systems, the vast expanse of airports, and the numerous angles captured by surveillance cameras, comprehensive monitoring is difficult for control room personnel. As airports expand in size, real-time monitoring of the area, particularly of individual objects around aircraft, becomes increasingly important. However, this approach is limited, with poor location recognition in areas with reduced visibility due to inclement weather or in areas further from the airfield.

[0005] Another common approach is to use surface surveillance radar to monitor aircraft and vehicles on the apron. However, this is expensive and unsuitable for smaller airports, making it only available at large and medium-sized hub airports. The most common approach involves operating surface surveillance radars independently, failing to integrate them with other advanced technologies.

[0006] Another common method now is to use electronic map technology, that is, to conduct geographical mapping of the airport area, then install positioning devices on the vehicles, and mark the vehicle's position, speed and direction on the electronic map based on the mapping.

[0007] The invention patent (application publication number CN114220297A, publication date 2022.03.22) discloses a video-based airport parking stand security monitoring method. The invention implements some content related to the monitoring work in local details, such as whether the inspection personnel are in place, whether the facilities and equipment are placed on time and accurately, and whether the ground service personnel are wearing reflective vests.

[0008] The invention patent (application number CN116403162A, publication date 2023.07.07) discloses a method for identifying target behaviors in airport scenes. The core function is still to identify the behaviors of typical non-cooperative targets in airport scenes by analyzing video streams.

[0009] The two aforementioned patents have significant shortcomings. They are still limited by weather conditions, unable to operate around the clock, and fail to comprehensively address monitoring issues. For example, they cannot identify aircraft in inclement weather and poor lighting conditions. They also fail to address the current situation of excessive manpower, dispersed monitoring personnel, 24 / 7 shift work, extreme cold and heat, and harsh environments with noise and exhaust. Advanced technologies have not yet been integrated into existing airport ground aircraft monitoring scenarios. Summary of the Invention

[0010] To overcome the problems existing in the related art, the disclosed embodiments of the present invention provide a panoramic aircraft monitoring system and method based on video stitching and electronic maps. Specifically, they relate to a panoramic video civil aviation airport aircraft monitoring management system, method, and electronic equipment based on the integration of video stitching and electronic maps. The purpose of the present invention is to solve the information communication problem in the production and operation of cargo stowage at civil aviation airports, and to provide a messaging platform system that meets business needs, is based on cargo stowage business rules, and supports text and images. The purpose of the present invention is to solve the problem of ground aircraft monitoring at civil aviation airports through technical means, and to provide a system that meets the needs of aircraft monitoring business, using AI algorithms based on high-definition camera video streams combined with electronic maps to create a panoramic view of the entire apron. This solves the problem of insufficient monitoring technology in inclement weather with limited visibility and provides a means of monitoring. The new technology is intended to allow security personnel to see the locations of personnel, vehicles, and aircraft through buildings and low clouds, providing an all-weather view that was previously impossible. This will reduce the working hours of monitoring personnel in adverse environments, reduce the workload of monitoring personnel, and improve work efficiency.

[0011] The technical solution is as follows: a panoramic aircraft monitoring method based on video splicing and electronic maps, the method comprising:

[0012] S1, the data interface processing server obtains vehicle and personnel location, aircraft and flight dynamic information from the positioning gateway server, and the video gateway obtains high-definition camera video stream data;

[0013] S2, the video stream processing server performs video splicing processing on the acquired high-definition camera video stream data;

[0014] S3, electronic map GIS server provides electronic map services;

[0015] S4: The panoramic video fusion server performs panoramic video fusion processing on the electronic map and the stitched video, calculates the aircraft's location information, and generates a panoramic video of people, vehicles, and aircraft based on the aircraft's location information. It also completes the display of personnel information, vehicle speeding alarms, electronic fences, boundary crossing alarms, and illegal intrusion alarms.

[0016] S5, the application server reads the received panoramic video alarm data and simultaneously completes personnel management, vehicle management, flight dynamic information management, and monitoring mode management;

[0017] S6, the APP client and the WEP client receive service and alarm information, and report normal monitoring and abnormal processing to the server through the client.

[0018] Furthermore, in step S2, the HD camera video stream is stitched using an AI algorithm to create a panoramic view of the entire apron.

[0019] Furthermore, in step S4, electronic map location data is integrated into the panoramic view through an AI algorithm. The specific process includes: mapping and converting GPS location information and pixel points to form a mapping table, so that the pixel points of the video splicing correspond one-to-one with the electronic map surveying points, and saving them to display the apron scene in real time.

[0020] Furthermore, in step S4, the vehicle is equipped with an auxiliary positioning device compatible with GPS and Beidou signals. The vehicle positioning data collected is used to draw the location information and vehicle information onto the panoramic view. In case of special circumstances or security level upgrades, the vehicle location information positioning frequency is increased and the refresh interval of the system front-end interface is reduced to realize the display of vehicle location information.

[0021] Vehicle information includes vehicle type, number, whether it is idle, whether it is a temporary construction vehicle, and speed, which can realize speeding warning and electronic fence warning;

[0022] If special circumstances occur or the security level is upgraded, the frequency of collecting vehicle location information and the time interval for refreshing the vehicle interface will be increased to display the vehicle location information.

[0023] In step S4, completing the hanging sign includes: the ID of the portable high-precision positioning terminal is unique, and the ID of the portable high-precision positioning terminal is bound to the wearer's personal information in the background of the panoramic fusion video system. During the front-end display process, the personnel information is reversely searched through the ID of the portable high-precision positioning terminal, and then the personnel information is displayed in the form of a hanging sign on the panoramic view personnel, vehicle, and aircraft images. The personnel information includes department, name, work number, and whether it is a temporary construction worker.

[0024] In step S4, the illegal intrusion alarm includes: analyzing the video stream data through the AI algorithm, identifying people and vehicles in the video stream, accumulating the people and vehicles respectively, and calculating the number of people and vehicles; when the calculated number of people and vehicles is inconsistent with the people and vehicle data on the electronic map, the location information of the people and vehicles is calculated through the mapping relationship between the panoramic video pixel points and the map coordinates, and the obtained location information is compared and analyzed with the location reported by the positioning device of the people and vehicles on the electronic map. If there are no people and vehicles at the corresponding position on the electronic map, the analyzed people and vehicles are regarded as external intrusion factors, and the system automatically marks the invading people and vehicles on the panoramic view and warns the on-duty personnel.

[0025] Furthermore, AI algorithms are used to analyze video stream data and identify people and vehicles in the video stream, including:

[0026] Step 1: The high-definition camera moves in a circle and continuously captures multiple frames of images of people, vehicles, and aircraft {P1, P2, P3, P4, ... P n}, select one of the aircraft images P3 as the reference frame;

[0027] Step 2: the image sequence {P1, P2, P3, P4, ... n}conduct times downsampling processing, u>1;

[0028] Step 3: For the image sequences {P1, P2, P3, P4, ... P n Perform Sobel edge detection and extract edges of image sequences of people, vehicles, and aircraft;

[0029] Step 4: Based on the detected edge pixel points of the personnel, vehicle and aircraft images, select the pixel points {a P =(a r ,b r)} is the central pixel block, for the person and vehicle images {P1,P2,P4,…P n} are matched pixel by pixel, and the distance judgment method is used to calculate the reference aircraft image P3 and

[0030] {P1,P2,P4,…P n Normalize the minimum value of the cross-correlation distance to obtain the matrix value H of the distance between n-1 corresponding edge pixels and the aircraft s (a), the calculation formula is:

[0031]

[0032] Step 5: Normalize and fit the distance matrix H(a) of the personnel and vehicle images to obtain P D The distance of the approaching aircraft to the stadium H D (a), and then H(a)-H D (a) Normalize the fitting again to obtain the distance motion field H of the target person approaching the aircraft F (a) Obtain the distance between the target vehicle and the aircraft after separation, and the distance between them and the distance between them. D (a) The distance between the target person and the aircraft is the distance between the target person and the aircraft. F (a); Parameter settings are as follows: number of iterations K: 1000-2000 times, deviation t: 5-10, decision threshold th: 5-9;

[0033] Step 6: Upsample the filtered images of people and vehicles by a factor of u to restore them to the size of the original image sequences of people and vehicles;

[0034] Step 7: Establish a feedforward neural network, use the AI algorithm to train the neural network sample data set, take two hidden layers, and use the Sigmoid transfer function for each layer of neurons. r b s ] and its corresponding H D (a) and H F (a) As sample data points for training neural networks, the relationship between input and output is as follows:

[0035]

[0036] Repeatedly train the neural network so that Among them H rs is the motion field value output by the neural network calculation, is the actual desired output motion field value, G and H are weighted vectors of each layer. After training, the network is generalized to other pixels to obtain the closest distance motion field O of the target vehicle image close to the aircraft.D (a, b) The closest distance between the target person and the aircraft is at the sports field O F (a,b);

[0037] Step 8: Adaptively filter the closest motion field to the aircraft and replace the pixel value with the median grayscale value Medmed in the pixel neighborhood. The calculation formula is:

[0038]

[0039] Step 9: Based on the closest distance motion field of the target person and vehicle images to the aircraft, coordinate registration is performed on the pixel points corresponding to the original reference person and vehicle images. The registered person and vehicle images are grayscaled, and each pixel is calculated. The RGB value of the corresponding position of the registered person and vehicle image with the middle grayscale value is selected as the new pixel value of the point, and the target vehicle image P after initial separation is restored. B and the target person image P F ;

[0040] Step 10: Based on the target vehicle image P after initial separation D and the target person image P F , optimization iteration min(P3-P D -P F ), according to P D +P F The normalized cross-correlation value of P3 is processed correctly, and the formula P3-P D -P F And set the threshold to refine the target vehicle image P D and the target person image P F The correct pixel point in the NCC, go to step 7, and continue with the subsequent steps. T >NCC T-1 , continue with the subsequent steps, calculate P each time iterate D +P F With the certified NCC value of P3, if NCC T >NCC T-1 , indicating that the accuracy of reconstructed personnel and vehicle images increases, continue iteration, otherwise, stop iteration and end the program.

[0041] Furthermore, in step S5, the monitoring mode includes:

[0042] In the non-defense mode, no alarm will be triggered for any person or vehicle in the designated area;

[0043] In the default defense mode, an alarm is issued for people and vehicles on the apron without positioning devices, and for those not wearing reflective clothing;

[0044] In the alarm deployment mode, an electronic fence is set up and an alarm will be triggered when anyone enters the electronic fence.

[0045] In step S5, the monitoring mode further includes: a single-aircraft monitoring mode and a multi-aircraft area monitoring mode.

[0046] Specifically, the unguarded mode includes: the aircraft monitoring system is in a non-operating state, does not monitor or detect aircraft on the apron and surrounding areas, and does not initiate identification and alarm for abnormal situations of unauthorized personnel or vehicles approaching;

[0047] The default defense mode includes: the aircraft monitoring system is armed according to the preset basic parameters and rules, the monitoring and protection functions are activated, and preliminary detection of abnormal conditions occurs;

[0048] The alert deployment mode includes: setting up an electronic fence to warn any incoming targets, using the aircraft monitoring system to monitor the video surveillance range and clarity, and adjusting or activating multiple monitoring and detection equipment to conduct all-round, real-time monitoring of the aircraft's surrounding environment. Alarms will be issued if there are abnormal behaviors of people or vehicles, potential security threats, or if the distance close to the aircraft reaches the set warning threshold;

[0049] The single-aircraft surveillance mode includes direct visual observation and supervision of the aircraft's surroundings, passengers embarking and disembarking, and cargo loading and unloading; and the use of surveillance cameras to focus the monitoring image on the aircraft for real-time video surveillance.

[0050] The multi-aircraft area monitoring mode includes: background monitoring through a video surveillance platform, combined with human and vehicle inspections and emergency response, management of multiple aircraft in the entire designated area, and comprehensive judgment and disposal of abnormal situations in the area.

[0051] Another object of the present invention is to provide a panoramic aircraft monitoring system based on video splicing and electronic maps, comprising:

[0052] The data interface processing server is used to obtain vehicle and personnel location, aircraft and flight dynamic information from the positioning gateway server, and obtain high-definition camera video stream data from the video gateway;

[0053] Video stream processing server, used for video splicing processing of acquired high-definition camera video stream data;

[0054] Electronic map GIS server, used to provide electronic map services;

[0055] The panoramic video fusion server is used to perform panoramic video fusion processing on the electronic map and the stitched video, calculate the aircraft position information, and generate a panoramic video of people, vehicles, and aircraft based on the aircraft position information. It also completes personnel information display, vehicle speeding alarm, electronic fence, cross-border alarm, and illegal intrusion alarm.

[0056] The application server is used to read the received panoramic video alarm data and simultaneously complete personnel management, vehicle management, flight dynamic information management, and monitoring mode management;

[0057] The client, including the APP client and the WEP client, receives service and alarm information, and reports normal monitoring and abnormal processing to the server.

[0058] In combination with all the above technical solutions, the beneficial effects of the present invention are as follows:

[0059] First, the present invention uses two complementary functional technologies, video fusion and electronic maps, to achieve panoramic video security monitoring of aircraft in different modes under all weather conditions. It can proactively detect intruding personnel and vehicles and issue alarms, which can reduce the input of manpower in the apron monitoring site. The present invention changes manual monitoring and problem discovery to automatic system monitoring and problem discovery, which significantly improves the work efficiency of monitoring personnel, reduces the workload of daily affairs, and shifts the focus of the overall monitoring personnel's work from busy information release and input to more reasonable planning and utilization of resources, thereby improving the overall collaborative work level and production efficiency. The present invention uses video splicing technology to automatically identify illegal personnel and vehicles and realize multiple monitoring modes.

[0060] Second, the present invention integrates two technical methods, which can automatically identify outsiders and vehicles, and solves the risks and problems that have long existed in the industry. Video splicing and electronic maps are two commonly used monitoring methods, but they are independently built and used. The present invention integrates two technical methods, which can automatically identify outsiders and vehicles, and support multiple monitoring modes. The panoramic video apron monitoring system that comprehensively utilizes the two technical methods of video splicing and electronic maps can automatically identify outsiders and vehicles and issue alarms, while supporting multiple monitoring modes. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure;

[0062] Figure 1 Schematic diagram of a panoramic aircraft monitoring system based on video splicing and electronic maps provided by an embodiment of the present invention;

[0063] Figure 2This is a flow chart of a panoramic aircraft monitoring method based on video splicing and electronic maps provided by an embodiment of the present invention;

[0064] Figure 3 This is a schematic diagram of a panoramic aircraft monitoring method based on video splicing and electronic maps provided by an embodiment of the present invention;

[0065] Figure 4 This is a simulation diagram of an experimental effect verification conducted at a small civil airport provided by an embodiment of the present invention;

[0066] In the figure: 1. Data interface processing server; 2. Video stream processing server; 3. Electronic map GIS server; 4. Panoramic video fusion server; 5. Application server; 6. Client. DETAILED DESCRIPTION

[0067] To make the above-mentioned objects, features, and advantages of the present invention more readily apparent, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings. The following description sets forth numerous specific details to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways than those described herein, and those skilled in the art may make similar modifications without departing from the scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0068] The panoramic aircraft monitoring system based on video splicing and electronic maps provided by the embodiments of the present invention can obtain vehicle location, personnel location, aircraft information, and flight dynamics information. The video gateway obtains high-definition camera video stream data; the acquired high-definition camera video stream data is processed by video splicing; the electronic map GIS server provides electronic map services; the electronic map and the spliced video are subjected to panoramic video fusion processing to calculate aircraft location information, and complete personnel management, vehicle management, flight dynamics information management, and monitoring mode management; normal monitoring and abnormal handling status are reported to the server through the client. The present invention uses the complementary functional technologies of video fusion and electronic mapping to achieve panoramic video security monitoring of aircraft in different modes under all weather conditions, and can actively detect intruding personnel and vehicles and issue alarms.

[0069] Example 1, as Figure 1 As shown, the panoramic aircraft monitoring system based on video splicing and electronic maps provided by the embodiment of the present invention includes:

[0070] The data interface processing server 1 is used to obtain vehicle and personnel location, aircraft and flight dynamic information from the positioning gateway server, and obtain high-definition camera video stream data from the video gateway.

[0071] The video stream processing server 2 is used to perform video splicing processing on the acquired high-definition camera video stream data.

[0072] The electronic map GIS server 3 is used to provide electronic map services.

[0073] The panoramic video fusion server 4 is used to perform panoramic video fusion processing on the electronic map and the spliced video, calculate the aircraft position information, and generate personnel, vehicles, and aircraft into a panoramic video according to the aircraft position information, and complete functions such as personnel information display, vehicle speeding alarm, electronic fence, cross-border alarm, illegal intrusion alarm, etc.

[0074] The application server 5 is used to read the received panoramic video alarm data and simultaneously complete functions such as personnel management, vehicle management, flight dynamic information management, and monitoring mode management.

[0075] The client 6, including the APP client and the WEP client, receives service and alarm information, and reports normal monitoring and abnormal processing to the server.

[0076] Example 2, as Figure 2 As shown, the panoramic aircraft monitoring method based on video splicing and electronic maps provided by the embodiment of the present invention includes:

[0077] S1, the data interface processing server obtains vehicle location, personnel location, aircraft and flight dynamic information from the positioning gateway server, and obtains high-definition camera video stream data from the video gateway.

[0078] S2, the video stream processing server performs video splicing processing on the acquired high-definition camera video stream data.

[0079] S3, the electronic map GIS server provides electronic map services, including tile services, location query services, etc.

[0080] In step S4, the panoramic video fusion server processes the electronic map and the stitched video, calculates aircraft location information, and generates a panoramic video of people, vehicles, and aircraft based on the aircraft location information. This server then implements functions such as personnel information signage, speeding warnings, electronic fencing, boundary crossing warnings, and illegal intrusion warnings. The illegal intrusion warning identification process is as follows: An AI algorithm analyzes the video stream data, identifies people and vehicles in the video stream, accumulates the number of people and vehicles, and calculates the number of people and vehicles. If the calculated number of people and vehicles does not match the number of people and vehicles on the electronic map, the server calculates their location information by mapping the panoramic video pixels to map coordinates. This location information is then compared with the positions reported by the positioning devices of people and vehicles on the electronic map. If no person or vehicle is found at the corresponding location on the electronic map, the identified person or vehicle is considered an intruder. The system automatically identifies the intruder in the panoramic view and issues an alert to the on-duty personnel.

[0081] It's understandable that existing technologies typically use single video stitching to achieve video processing, or use electronic maps to locate people, vehicles, and aircraft. However, they haven't integrated or applied video stitching and electronic maps. The present invention achieves this functionality by integrating video stitching and electronic maps.

[0082] Moreover, the present invention maps the pixel points of the spliced video and the position coordinate information of the map, and draws the road network data of the map into the panoramic video according to the mapping relationship, and draws the vehicle, personnel and aircraft icons into the panoramic video in real time, completing the functions of personnel information display, vehicle speeding alarm, electronic fence, cross-border alarm, illegal intrusion alarm, etc.

[0083] Vehicle registration displays information such as the license plate number on the vehicle icon. The principle of the vehicle speeding alarm is that vehicles on the apron cannot exceed the speed limit set by the airport. The electronic fence is an area demarcated on an electronic map based on needs. The cross-border alarm is triggered when people or vehicles without permission enter the electronic fence system. The illegal entry alarm is triggered when the system automatically identifies illegal people and vehicles.

[0084] S5, the application server reads the received panoramic video alarm data and simultaneously completes functions such as personnel management, vehicle management, flight dynamic information management, and monitoring mode management.

[0085] S6, the APP client and the WEP client receive service and alarm information, and report normal monitoring and abnormal processing to the server through the client.

[0086] Among them, the video stitching in step S2 and the panoramic video fusion in step S4 are both fused through the AI algorithm provided by the AI algorithm server.

[0087] In step S3, the electronic map GIS server provides electronic map services, performs professional mapping of the airport, and obtains a high-precision electronic map of the airport. The electronic map accuracy is 1 meter, and higher precision, i.e., centimeter-level mapping, can be achieved as needed.

[0088] In step S4, the video stream data is analyzed by an AI algorithm, and the identification of people and vehicles in the video stream specifically includes:

[0089] Step 1: The high-definition camera moves in a circle and continuously captures multiple frames of images of people, vehicles, and aircraft {P1, P2, P3, P4, ... P n}, select one of the aircraft images P3 as the reference frame;

[0090] Step 2: the image sequence {P1, P2, P3, P4, ... n}conduct times downsampling processing, u>1;

[0091] Step 3: For the image sequences {P1, P2, P3, P4, ... P n Perform Sobel edge detection and extract edges of image sequences of people, vehicles, and aircraft;

[0092] Step 4: Based on the detected edge pixel points of the personnel, vehicle and aircraft images, select the pixel points {a P =(a r ,b r )} is the central pixel block, for the person and vehicle images {P1,P2,P4,…P n} are matched pixel by pixel, and the distance judgment method is used to calculate the reference aircraft image P3 and

[0093] {P1,P2,P4,…P n Normalize the minimum value of the cross-correlation distance to obtain the matrix value H of the distance between n-1 corresponding edge pixels and the aircraft s (a), the calculation formula is:

[0094]

[0095] Step 5: Normalize and fit the distance matrix H(a) of the personnel and vehicle images to obtain P D The distance of the approaching aircraft to the stadium H D (a), and then H(a)-H D(a) Normalize the fitting again to obtain the distance motion field H of the target person approaching the aircraft F (a) Obtain the distance between the target vehicle and the aircraft after separation, and the distance between them and the distance between them. D (a) The distance between the target person and the aircraft is the distance between the target person and the aircraft. F (a); Parameter settings are as follows: number of iterations K: 1000-2000 times, deviation t: 5-10, decision threshold th: 5-9;

[0096] Step 6: Upsample the filtered images of people and vehicles by a factor of u to restore them to the size of the original image sequences of people and vehicles;

[0097] Step 7: Establish a feedforward neural network, use the AI algorithm to train the neural network sample data set, take two hidden layers, and use the Sigmoid transfer function for each layer of neurons. r b s ] and its corresponding H D (a) and H F (a) As sample data points for training neural networks, the relationship between input and output is as follows:

[0098]

[0099] Repeatedly train the neural network so that Among them H rs is the motion field value output by the neural network calculation, is the actual desired output motion field value, G and H are weighted vectors of each layer. After training, the network is generalized to other pixels to obtain the closest distance motion field O of the target vehicle image close to the aircraft. D (a, b) The closest distance between the target person and the aircraft is at the sports field O F (a,b);

[0100] Step 8: Adaptively filter the closest motion field to the aircraft and replace the pixel value with the median grayscale value Medmed in the pixel neighborhood. The calculation formula is:

[0101]

[0102] Step 9: Based on the closest distance motion field of the target person and vehicle images to the aircraft, coordinate registration is performed on the pixel points corresponding to the original reference person and vehicle images. The registered person and vehicle images are grayscaled, and each pixel is calculated. The RGB value of the corresponding position of the registered person and vehicle image with the middle grayscale value is selected as the new pixel value of the point, and the target vehicle image P after initial separation is restored. Band the target person image P F ;

[0103] Step 10: Based on the target vehicle image P after initial separation D and the target person image P F , optimization iteration min(P3-P D -P F ), according to P D +P F The normalized cross-correlation value of P3 is processed correctly, and the formula P3-P D -P F And set the threshold to refine the target vehicle image P D and the target person image P F The correct pixel point in the NCC, go to step 7, and continue with the subsequent steps. T >NCC T-1 , continue with the subsequent steps, calculate P each time iterate D +P F With the certified NCC value of P3, if NCC T >NCC T-1 , indicating that the accuracy of reconstructed personnel and vehicle images increases, continue iteration, otherwise, stop iteration and end the program.

[0104] The present invention further refines the target vehicle image and the target person image by continuously optimizing the interpolated closest distance motion field of the approaching aircraft and the closest distance motion field of the target person approaching the aircraft.

[0105] An AI algorithm is used to fit the motion field of the entire person and vehicle image. This motion field is then used to coordinately register multiple frames of captured person and vehicle images, reconstructing two separate layers of person and vehicle images. Optimization and iteration are then performed on the initially separated person and vehicle images to further refine the details of the separated images. The method described in this invention can significantly improve the distance between the target person, vehicle, and aircraft image when processing images of severely dangerous targets.

[0106] While achieving comparable results to previous methods, this method avoids the requirement for uniformly spaced sampling and reduces image capture restrictions. Downsampling and upsampling high-resolution personnel, vehicle, and aircraft images before edge extraction effectively reduces the time required to calculate the normalized cross-correlation between adjacent personnel and vehicle images. Without downsampling, calculating the NCC between a reference personnel or vehicle image and an adjacent personnel or vehicle image takes 300 seconds. With downsampling, this method reduces this time to 15 seconds, significantly improving speed. The near-aircraft distance between the reference personnel or vehicle image and multiple adjacent personnel or vehicle images is simultaneously calculated, providing a more accurate method for subsequent coordinate matching. Using a sequence of personnel, vehicle, and aircraft images with a resolution of 1152×648 as an example, this method achieved a total time of one minute using an Intel(R) Core(TM) i5 CPU processor and 8GB of RAM on a PC.

[0107] In step S4, in the panoramic view, the electronic map location data is integrated to make the pixel points correspond one-to-one with the electronic map mapping points, and the dynamic and static scenes of the apron are displayed in real time, and a 3D dynamic effect is achieved through the AI algorithm.

[0108] For example, in a panoramic view, electronic map location data is integrated through AI algorithms. The specific process includes: mapping and converting GPS location information and pixel points to form a mapping table, so that the pixel points of the video splicing correspond one-to-one with the electronic map surveying points, and saving them to display the apron scene in real time.

[0109] In step S4, the vehicle is installed with an auxiliary positioning device that is compatible with GPS and Beidou signals, and improves the vehicle positioning accuracy through differential positioning base stations. The vehicle positioning data collected is used to draw vehicle information on the panoramic view. Specific information includes vehicle type, number, whether it is idle (if performing a security mission, the flight number of the service is displayed), whether it is a temporary construction vehicle, and speed. Speeding warnings and electronic fence warnings can be implemented. If special circumstances are encountered or the security level is upgraded, the frequency of vehicle location information collection and the time interval for vehicle interface refresh can be increased to achieve a more accurate display of vehicle location information.

[0110] For example, the vehicle is installed with an auxiliary positioning device compatible with GPS and Beidou signals, and the location information and vehicle information are drawn onto a panoramic view through the collected vehicle positioning data; if special circumstances (special sensitive periods, threat intelligence warnings, emergencies, etc.) and security level upgrades (such as during major events, specific flight support, etc.) are encountered, the vehicle location information positioning frequency is increased (such as the frequency from 1 time / second to 10 times / second), and the refresh time interval of the system front-end interface is reduced to achieve a more accurate display of vehicle location information.

[0111] In step S4, the system implements the posting of staff information. Staff members wear portable high-precision positioning terminals that support indoor UWB positioning and outdoor Beidou positioning, and communicate data through 5G public networks or 5G private networks.

[0112] Exemplarily, in step S4, completing the hanging sign specifically includes: the ID of the portable high-precision positioning terminal is unique, and the ID of the portable high-precision positioning terminal is bound to the wearer's personal information in the background of the panoramic fusion video system. During the front-end display process, the personnel information is reversely searched through the ID of the portable high-precision positioning terminal, and then the personnel information is displayed in the form of a hanging sign on the panoramic view personnel, vehicle, and aircraft images. The personnel information includes department, name, work number, and whether it is a temporary construction worker.

[0113] In step S4, the system analyzes and processes the images of personnel, vehicles, and aircraft, identifies personnel who are not wearing reflective clothing, marks them in the panoramic view, and determines whether to issue an alarm based on the current monitoring mode.

[0114] In step S4, the illegal intrusion alarm includes: analyzing the video stream data through the AI algorithm, identifying people and vehicles in the video stream, accumulating the people and vehicles respectively, and calculating the number of people and vehicles; when the calculated number of people and vehicles is inconsistent with the people and vehicle data on the electronic map, the location information of the people and vehicles is calculated through the mapping relationship between the panoramic video pixel points and the map coordinates, and the obtained location information is compared and analyzed with the location reported by the positioning device of the people and vehicles on the electronic map. If there are no people and vehicles at the corresponding position on the electronic map, the analyzed people and vehicles are regarded as external intrusion factors, and the system automatically marks the invading people and vehicles on the panoramic view and warns the on-duty personnel.

[0115] The present invention realizes all-weather monitoring of aircraft on the apron through the complementary technologies of the above two methods and can effectively reduce manpower input.

[0116] In step S5, specifically, the supervisor is managed, including inspection records, emergency response, etc.

[0117] In the monitoring mode management, the monitoring of airport aircraft is divided into three operating modes: unguarded mode (no alarm is triggered for any behavior, personnel, or vehicle in the designated area, which is generally the case when boarding an aircraft at a remote parking position), default armed mode (alarms are triggered for personnel and vehicles on the apron without positioning devices, and alarms are triggered for those not wearing reflective clothing), and alert deployment mode (an electronic fence is set up, and anyone entering the electronic fence will be alarmed). In addition, it also supports single-aircraft monitoring mode (the monitoring range is a 30-meter area around a single aircraft, and the documents of personnel getting on and off the aircraft and suspicious personnel and vehicles approaching the aircraft are checked; the documents of personnel and vehicles entering the monitoring area are checked; and personnel and items that have not passed security inspections are prevented from entering the aircraft) and multi-aircraft area monitoring mode (the apron is divided into several areas, each of which can accommodate multiple aircraft. The area monitoring combines human defense with technical defense, that is, fixed guard posts, patrol inspection posts, and video monitoring posts are effectively linked to conduct full-range three-dimensional monitoring of various abnormal situations in the apron area. At the same time, it is responsible for monitoring multiple aircraft, and the scope of safety control is greatly improved).

[0118] Exemplarily, technical terms related to the present invention further include:

[0119] Unarmed mode.

[0120] No alerts are issued for any activity, personnel, or vehicles on the apron within the controlled area. In this mode, the aircraft monitoring system is essentially inactive, performing no monitoring, inspection, or other protective operations on the aircraft and surrounding areas. It also does not activate its identification and alert functions for abnormal situations, such as unauthorized personnel or vehicles approaching. This is typically used in special circumstances or for specific debugging and maintenance scenarios, such as large-scale facility and equipment renovations or system upgrades at airports, when the monitoring system's protective functions need to be temporarily disabled to facilitate construction work.

[0121] Default defense mode.

[0122] Within the control area, warnings are issued for personnel and vehicles on the apron without positioning devices, and for those not wearing reflective clothing. In this mode, the aircraft monitoring system is set up according to preset basic parameters and rules, usually activating some basic monitoring and protection functions, such as regular video surveillance within a certain range around the aircraft and preliminary detection of obvious abnormalities that may occur. Generally, it can provide the most basic safety protection for aircraft, ensuring preliminary prevention of some common and relatively obvious safety risks in daily operations, such as preventing general unrelated personnel from accidentally entering the area near the aircraft. This is a normalized and basic protection mode that can ensure the basic safety of aircraft in most cases.

[0123] Alarm arming mode.

[0124] Set up an electronic fence, and anyone entering the electronic fence will be alarmed. In this mode, the aircraft monitoring system will increase the level, sensitivity, and frequency of monitoring and prevention. In addition to enhancing the scope and clarity of video surveillance, more manual monitoring and detection equipment may be adjusted or enabled, such as infrared sensors, radar monitoring, manual patrols, and personnel on duty, to conduct all-round, real-time monitoring of the aircraft's surrounding environment. Once a person, vehicle, etc. is found to be exhibiting obvious abnormal behavior or the distance approaching the aircraft reaches the set warning threshold, the system will immediately issue an alarm. It is generally used to respond to situations with potentially higher security risks, such as when there are important events at the airport, during sensitive periods, or when intelligence of possible security threats is received. It can promptly detect and warn of potential security threats, buying more time for security personnel to respond and take action, and maximizing the safety of aircraft and airports.

[0125] Single aircraft tracking mode.

[0126] In this mode, dedicated monitoring personnel or equipment are arranged to closely monitor a single aircraft throughout its entire journey. The monitoring personnel will remain stationed near the aircraft during its docking, and conduct direct visual observation and supervision of the aircraft's surrounding environment, passengers boarding and disembarking, cargo loading and unloading, etc. At the same time, dedicated surveillance cameras will focus the monitoring screen on this aircraft for real-time video monitoring. It is generally used in some special aircraft security situations, such as flights carrying important people, special materials, or flights with higher security risk assessments. It can provide the most direct and focused security protection, ensuring that every link of the aircraft during its docking is under strict monitoring to prevent any possible security incidents.

[0127] Multi-aircraft area monitoring mode.

[0128] Under this model, the airport's parking area is divided into multiple surveillance zones, each with established posts for video surveillance, fixed-duty personnel, patrol inspections, and emergency response. Backstage monitoring is conducted via a video surveillance platform, combined with human and vehicle patrols and emergency response, to achieve comprehensive management of multiple aircraft within the designated area. The flow of personnel and vehicles within the zone can be uniformly monitored and managed, and abnormal situations within the zone can be comprehensively assessed and addressed. This model has high requirements for various airport conditions and is generally only adopted by large airports. It can improve the airport's overall surveillance efficiency and resource utilization, avoiding the potential waste of manpower associated with a single-aircraft surveillance model. While expanding the scope of security and control across the entire apron, it achieves a comprehensive, three-dimensional security control system that combines points and surfaces. Technically, it achieves full coverage of the aircraft surveillance zone, effectively addressing the dense flow of flights and frequent personnel and vehicle movements at airports, and comprehensively elevating the airport's air defense safety management capabilities.

[0129] For example, the principle of the panoramic aircraft monitoring method based on video splicing and electronic map provided by the embodiment of the present invention is as follows: Figure 3 shown.

[0130] To further illustrate the effects of the embodiments of the present invention, the following experiments were conducted.

[0131] After the development of the panoramic aircraft monitoring system was completed, the present invention selected a small civil airport for experimentation to verify its effectiveness. A single-aircraft monitoring mode was selected, requiring ample light and clear weather, good GPS / Beidou coverage, and accurate positioning accuracy. A designated apron area was designated, with a tester and a test vehicle located within the area. The panoramic aircraft monitoring system was activated. In scenario one, the tester turned on their portable GPS / Beidou receiver, and the test vehicle turned on its GPS / Beidou positioning terminal. At this point, the tester and vehicle were visible on both the electronic map and the video surveillance. The panoramic aircraft monitoring system operated normally, correctly identifying the tester and vehicle and accurately mapping them into the fused panoramic video. No system alarms were issued. In scenario two, the tester turned off their portable GPS / Beidou receiver, and the test vehicle turned off its GPS / Beidou positioning terminal. At this point, the tester and vehicle were no longer visible on the electronic map, but were visible on the video surveillance. After 5 seconds, the system accurately identified the tester and vehicle's locations, deemed them to be illegal intrusions, and issued an alarm.

[0132] After analysis, it was found that the 5-second delay was due to the alarm lag caused by video stitching, receiving positioning data, generating panoramic videos and data processing. In actual scenarios, a 5-second delay is still acceptable. In addition, by optimizing the processing business logic and using better-performing hardware equipment, the recognition and alarm time can be shortened to a certain extent.

[0133] After communicating with airport surveillance personnel, users believe that the system can use technical means to identify illegal intruders and vehicles, support multiple surveillance modes, effectively improve work quality, and reduce work pressure. They expressed their approval of the system and believe that it can be promoted in civil aviation.

[0134] The above description is only a preferred specific implementation method of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with this technical field within the technical scope disclosed by the present invention and within the spirit and principles of the present invention should be covered by the scope of protection of the present invention.

Claims

1. A panoramic aircraft monitoring method based on video splicing and electronic maps, characterized in that: The method comprises the following steps: S1, the data interface processing server obtains vehicle location information, personnel location information, aircraft location and flight dynamic information from the positioning gateway server, and obtains high-definition camera video stream data from the video gateway; S2, the video stream processing server performs video splicing processing on the acquired high-definition camera video stream data; S3, electronic map GIS server provides electronic map services; In step S4, the panoramic video fusion server performs panoramic video fusion processing on the electronic map and the stitched video, calculates the aircraft's location information, and generates a panoramic video of people, vehicles, and aircraft. It also provides personnel information signage, vehicle speeding alarms, electronic fences, boundary crossing alarms, and illegal intrusion alarms. S5, the application server reads the received panoramic video alarm data and completes personnel management, vehicle management, flight dynamic information management, and monitoring mode management; S6, the APP client and the WEP client receive service and alarm information, and report normal monitoring and abnormal processing to the server through the client.

2. The panoramic aircraft monitoring method based on video splicing and electronic maps according to claim 1 is characterized in that: In step S2, the HD camera video streams are stitched together using AI algorithms to create a panoramic view of the entire apron.

3. The panoramic aircraft monitoring method based on video splicing and electronic maps according to claim 2 is characterized in that: In the panoramic view, electronic map location data is integrated; The specific process includes: mapping and converting GPS location information and pixel points to form a mapping table, so that the pixel points of the video splicing correspond one-to-one with the electronic map surveying points, and saving them to display the apron scene in real time.

4. The panoramic aircraft monitoring method based on video splicing and electronic maps according to claim 1 is characterized in that: In step S4, the vehicle is equipped with an auxiliary positioning device compatible with GPS and Beidou signals. The location information and vehicle information are mapped onto the panoramic view using the collected vehicle positioning data. In special circumstances or when the security level is upgraded, the vehicle location information is displayed by increasing the positioning frequency and reducing the refresh interval of the system front-end interface. Vehicle information includes vehicle type, number, whether it is idle, whether it is a temporary construction vehicle, and speed, so as to realize speeding warning and electronic fence warning.

5. The panoramic aircraft monitoring method based on video splicing and electronic maps according to claim 1 is characterized in that: In step S4, completing the hanging sign includes: binding the ID of the portable high-precision positioning terminal with the wearer's personal information in the background of the panoramic fusion video system; in the front-end display process, reversely searching for personnel information through the ID of the portable high-precision positioning terminal; and then displaying the personnel information in the form of a hanging sign on the panoramic view personnel, vehicle, and aircraft images. The personnel information includes department, name, work number, and whether it is a temporary construction worker.

6. The panoramic aircraft monitoring method based on video splicing and electronic maps according to claim 1 is characterized in that: In step S4, the illegal intrusion alarm includes: analyzing the video stream data through the AI algorithm, identifying people and vehicles in the video stream, accumulating the people and vehicles respectively, and calculating the number of people and vehicles; when the calculated number of people and vehicles is inconsistent with the people and vehicle data on the electronic map, the location information of the people and vehicles is calculated through the mapping relationship between the panoramic video pixel points and the map coordinates, and the obtained location information is compared and analyzed with the location reported by the positioning device of the people and vehicles on the electronic map. If there are no people and vehicles at the corresponding position on the electronic map, the analyzed people and vehicles are regarded as external intrusion factors, and the system automatically marks the invading people and vehicles on the panoramic view and warns the on-duty personnel.

7. The panoramic aircraft monitoring method based on video splicing and electronic maps according to claim 6, characterized in that: The AI algorithm analyzes video stream data and identifies people and vehicles in the video stream, including: Step 1: The high-definition camera moves in a circle and continuously captures multiple frames of images of people, vehicles, and aircraft {P1, P2, P3, P4, ... P n }, select one of the aircraft images P3 as the reference frame; Step 2: the image sequence {P1, P2, P3, P4, ... n }conduct times downsampling processing, u>1; Step 3: For the image sequences of people, vehicles and aircraft {P1, P2, P3, P4, ... n Perform Sobel edge detection and extract edges of image sequences of people, vehicles, and aircraft; Step 4: Based on the detected edge pixel points of the personnel, vehicle and aircraft images, select the pixel points {a P =(a r ,b r )} is the central pixel block, for the person and vehicle images {P1,P2,P4,…P n } are matched pixel by pixel, and the distance judgment method is used to calculate the distance between the reference aircraft image P3 and {P1, P2, P4, ... n Normalize the minimum value of the cross-correlation distance to obtain the matrix value H of the distance between n-1 corresponding edge pixels and the aircraft s (a), the calculation formula is: Step 5: Normalize and fit the distance matrix H(a) of the personnel and vehicle images to obtain P D The distance of the approaching aircraft to the stadium H D (a), and then H(a)-H D (a) Normalize the fitting again to obtain the distance motion field H of the target person approaching the aircraft F (a) Obtain the distance between the target vehicle and the aircraft after separation, and the distance between them and the distance between them. D (a) The distance between the target person and the aircraft is the distance between the target person and the aircraft. F (a); Parameter settings are as follows: number of iterations K: 1000-2000 times, deviation t: 5-10, decision threshold th: 5-9; Step 6: Upsample the filtered images of people and vehicles by a factor of u to restore them to the size of the original image sequences of people and vehicles; Step 7: Establish a feedforward neural network, use the AI algorithm to train the neural network sample data set, take two hidden layers, and use the Sigmoid transfer function for each layer of neurons. r b s ] and its corresponding H D (a) and H F (a) As sample data points for training neural networks, the relationship between input and output is as follows: Repeatedly train the neural network so that Among them H rs is the motion field value output by the neural network calculation, is the actual desired output motion field value, G and H are weighted vectors of each layer. After training, the network is generalized to other pixels to obtain the closest distance motion field O of the target vehicle image close to the aircraft. D (a, b) The closest distance between the target person and the aircraft is at the sports field O F (a,b); Step 8: Adaptively filter the closest motion field to the aircraft and replace the pixel value with the median grayscale value Medmed in the pixel neighborhood. The calculation formula is: Step 9: Based on the closest distance motion field of the target person and vehicle images to the aircraft, coordinate registration is performed on the pixel points corresponding to the original reference person and vehicle images. The registered person and vehicle images are grayscaled, and each pixel is calculated. The RGB value of the corresponding position of the registered person and vehicle image with the middle grayscale value is selected as the new pixel value of the point, and the target vehicle image P after initial separation is restored. B and the target person image P F ; Step 10: Based on the target vehicle image P after initial separation D and the target person image P F , optimization iteration min(P3-P D -P F ), according to P D +P F The normalized cross-correlation value of P3 is processed correctly, and the formula P3-P D -P F And set the threshold to refine the target vehicle image P D and the target person image P F If the correct pixel in NCC is T >NCC T-1 , continue with the subsequent steps, calculate P each time iterate D +P F With the certified NCC value of P3, if NCC T >NCC T-1 , indicating that the accuracy of reconstructed personnel and vehicle images increases, continue iteration, otherwise, stop iteration and end the program.

8. The panoramic aircraft monitoring method based on video splicing and electronic maps according to claim 1, characterized in that: In step S5, the monitoring modes include: unarmed mode, default armed mode, alert armed mode, single aircraft monitoring mode and multi-aircraft area monitoring mode.

9. The panoramic aircraft monitoring method based on video splicing and electronic maps according to claim 8, characterized in that: The unguarded mode includes: the aircraft monitoring system is in a non-operating state, does not monitor or detect aircraft on the apron and surrounding areas, and does not initiate identification and alarm for abnormal approaching situations of unrelated personnel or vehicles; The default defense mode includes: the aircraft monitoring system is armed according to the preset basic parameters and rules, the monitoring and protection functions are activated, and preliminary detection of abnormal conditions occurs; The alert deployment mode includes: setting up an electronic fence to warn any incoming targets, using the aircraft monitoring system to monitor the video surveillance range and clarity, and adjusting or activating multiple monitoring and detection equipment to conduct all-round, real-time monitoring of the aircraft's surrounding environment. Alarms will be issued if there are abnormal behaviors of people or vehicles, potential security threats, or if the distance close to the aircraft reaches the set warning threshold; The single-aircraft surveillance mode includes direct visual observation and supervision of the aircraft's surroundings, passengers embarking and disembarking, and cargo loading and unloading; and the use of surveillance cameras to focus the monitoring image on the aircraft for real-time video surveillance. The multi-aircraft area monitoring mode includes: background monitoring through a video surveillance platform, combined with human and vehicle inspections and emergency response, management of multiple aircraft in the entire designated area, and comprehensive judgment and disposal of abnormal situations in the area.

10. A panoramic aircraft monitoring system based on video splicing and electronic maps, characterized in that: The system implements the panoramic aircraft monitoring method based on video splicing and electronic maps as described in any one of claims 1 to 9, and the system includes: A data interface processing server (1) is used to obtain vehicle and personnel location, aircraft and flight dynamic information from a positioning gateway server, and obtain high-definition camera video stream data from a video gateway; A video stream processing server (2) is used for performing video splicing processing on the acquired high-definition camera video stream data; Electronic map GIS server (3), used to provide electronic map services; A panoramic video fusion server (4) is used to perform panoramic video fusion processing on the electronic map and the spliced video, calculate the aircraft position information, and generate a panoramic video of people, vehicles, and aircraft according to the aircraft position information, and complete personnel information display, vehicle speeding alarm, electronic fence, crossing the boundary alarm, and illegal intrusion alarm; An application server (5) is used to read the received panoramic video alarm data and simultaneously complete personnel management, vehicle management, flight dynamic information management, and monitoring mode management; The client (6), including the APP client and the WEP client, receives service and alarm information, and reports normal monitoring and abnormal processing to the server.

Images