Airport panoramic monitoring system and airport panoramic monitoring method

Abstract

The present application discloses an airport panoramic monitoring system and an airport panoramic monitoring method, and relates to the field of monitoring technology. The system includes a radar module, a panoramic video monitoring module and a central control module, and the central control module is respectively connected to the radar module and the panoramic video monitoring module in communication; the radar module is used to obtain the radar data of the current airport, and process the radar data to obtain the position data of each target object in the airport, and send the position data of each target object to the central control module; the central control module is used to convert the position data of each target object, and synchronously output the converted position data of each target object to the panoramic video monitoring module; the panoramic video monitoring module is used to obtain the video image of the current airport, and mark each target object in the video image according to the converted position data of each target object. In this way, accurate and comprehensive monitoring of the airport can be achieved in rainy, foggy and other weather conditions.

Description

Technical Field

[0001] The present application relates to the field of monitoring technology, and in particular to an airport panoramic monitoring system and an airport panoramic monitoring method. Background Art

[0002] The airport panoramic monitoring system is a platform that can provide continuous, all-round and real-time airport monitoring. It collects real-time data such as images, sounds, and temperature of the airport by installing cameras or sensors at various locations in the airport. It then processes and analyzes this data to provide airport control personnel with complete airport information, thereby facilitating their decision-making and management.

[0003] However, in rainy, foggy and other weather conditions, the camera's line of sight may be obstructed, resulting in visual blind spots, thereby limiting the visible range and image clarity of the airport's panoramic monitoring system. In addition, fog, raindrops, lighting, etc. will also affect the images captured by the camera, making it impossible for airport control personnel to obtain accurate and comprehensive image information. Summary of the Invention

[0004] The main purpose of this application is to provide an airport panoramic monitoring system and an airport panoramic monitoring method to achieve accurate and comprehensive monitoring of the airport under rainy, foggy and other weather conditions.

[0005] To achieve the above-mentioned object, the present application provides an airport panoramic monitoring system, comprising a radar module, a panoramic video monitoring module and a central control module, wherein the central control module is communicatively connected to the radar module and the panoramic video monitoring module respectively;

[0006] The radar module is used to obtain radar data of the current airport, perform data processing on the radar data, obtain position data of each target object in the airport, and send the position data of each target object to the central control module;

[0007] The central control module is used to convert the position data of each target object and synchronously output the converted position data of each target object to the panoramic video monitoring module;

[0008] The panoramic video monitoring module is used to obtain the current video image of the airport and mark each target object in the video image according to the converted position data of each target object.

[0009] Optionally, the radar module is specifically used to: convert the radar data into a radar image, and obtain the grayscale value change rate of each pixel in the radar image; when the grayscale value change rate of the pixel is less than a preset threshold, determine the pixel as a noise point; and denoise the noise point according to the sign of the grayscale value change rate of the noise point.

[0010] Optionally, the position data of the target object includes the polar coordinate data of the target object, and the central control module includes middleware. The central control module is specifically used to: perform coordinate conversion on the polar coordinate data of each target object to obtain the intermediate coordinate data of each target object in the coordinate system of the panoramic video surveillance module; use the middleware to convert the data format of the intermediate coordinate data of each target object into a target data format, and the target data format is the data format corresponding to the panoramic video surveillance module; and synchronously transmit the intermediate coordinate data of each target object after data format conversion to the panoramic video surveillance module.

[0011] Optionally, the panoramic video surveillance module is specifically used to: determine the geographic coordinate data of each target object based on the intermediate coordinate data of each target object; perform coordinate conversion on the geographic coordinate data of each target object to obtain the plane coordinate data of each target object, where the plane coordinate data of the target object is the position coordinate of the target object in the coordinate system of the video image; and mark each target object in the video image based on the plane coordinate data of each target object.

[0012] Optionally, the radar module is further specifically used to obtain basic information of each target object and send the basic information of each target object to the central control module; the central control module is further specifically used to synchronously send the basic information of each target object to the panoramic video monitoring module; the panoramic video monitoring module is further specifically used to use the circle function and according to the plane coordinate data of each target object, to perform graphic annotation on each target object in the video image, and to use the putText function and according to the basic information of each target object, to perform text annotation on each target object in the video image.

[0013] Optionally, the radar module includes at least one radar, and the panoramic video surveillance module includes at least one camera device; each radar is used to obtain the radar data of the current airport; each camera device is used to obtain the video image of the current airport; the central control module is also used to perform time starting point alignment and data frame synchronization processing on each radar and each camera device.

[0014] Optionally, the panoramic video monitoring module further includes a display, and the display is used to display the video image after the graphic annotation and the text annotation.

[0015] In addition, to achieve the above-mentioned purpose, the present application also provides an airport panoramic monitoring method, which is applied to the airport panoramic monitoring system as described above, and the method includes: the panoramic video monitoring module obtains the current video image of the airport; the radar module obtains the radar data of the current airport, and processes the radar data to obtain the position data of each target object in the airport, and sends the position data of each target object to the central control module; the central control module converts the position data of each target object, and synchronously outputs the converted position data of each target object to the panoramic video monitoring module; the panoramic video monitoring module marks each target object in the video image according to the converted position data of each target object.

[0016] Optionally, the radar data is processed to obtain the position data of each target object in the airport, including: converting the radar data into a radar image, and obtaining the grayscale value change rate of each pixel point in the radar image; when the grayscale value change rate of the pixel point is less than a preset threshold, determining the pixel point as a noise point; and denoising the noise point according to the sign of the grayscale value change rate of the noise point.

[0017] Optionally, the position data of the target object includes the polar coordinate data of the target object; the data conversion of the position data of each target object includes: performing coordinate conversion on the polar coordinate data of each target object to obtain the intermediate coordinate data of each target object in the coordinate system of the panoramic video surveillance module; converting the data format of the intermediate coordinate data of each target object into a target data format, and the target data format is the data format corresponding to the panoramic video surveillance module; and synchronously transmitting the intermediate coordinate data of each target object after the data format conversion to the panoramic video surveillance module.

[0018] Optionally, the method of marking each target object in the video image based on the converted position data of each target object includes: determining the geographic coordinate data of each target object based on the intermediate coordinate data of each target object; performing coordinate conversion on the geographic coordinate data of each target object to obtain the plane coordinate data of each target object, the plane coordinate data of the target object being the position coordinates of the target object in the coordinate system of the video image; and marking each target object in the video image based on the plane coordinate data of each target object.

[0019] Optionally, the panoramic video monitoring module converts the geographic coordinate data of the target object into the plane coordinate data of the target object using the following formula:

[0020]

[0021] in, is the plane coordinate data of the target object, R is the radius of the earth, λ is the longitude in the geographic coordinate data of the target object, and φ is the latitude in the geographic coordinate data of the target object.

[0022] Optionally, after the radar module processes the radar data to obtain the position data of each target object in the airport, the method further includes: the radar module obtains basic information of each target object and sends the basic information of each target object to the central control module; the central control module synchronously sends the basic information of each target object to the panoramic video monitoring module; and according to the plane coordinate data of each target object, each target object is marked in the video image, including: using the circle function and according to the plane coordinate data of each target object, each target object is graphically marked in the video image, and using the putText function and according to the basic information of each target object, each target object is textually marked in the video image.

[0023] The present application also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for panoramic monitoring of an airport as described above is implemented.

[0024] The present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for panoramic monitoring of an airport as described above is implemented.

[0025] The airport panoramic monitoring system of the embodiment of the present application uses a radar module to perform real-time scanning of the airport to obtain radar data, and processes the radar data to obtain position data of each target object in the airport; the central control module converts the position data of each target object and synchronously outputs the converted position data of each target object to the panoramic video monitoring module; the panoramic monitoring module obtains a video image of the current airport, and annotates each target object in the video image based on the converted position data of each target object. Thus, in rainy, foggy and other weather conditions, the radar module is used to accurately detect the target objects at the airport, the radar data and the video monitoring data are combined, and the target objects monitored by the radar are visually annotated in the video image, so that airport staff can obtain airport monitoring information more comprehensively and accurately, thereby realizing accurate and comprehensive monitoring of the airport. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is an example scenario of the airport panoramic monitoring system provided by this application;

[0027] Figure 2 Schematic diagram of the structure of the airport panoramic monitoring system according to an embodiment of the present application;

[0028] Figure 3 This is one of the flow charts of the airport panoramic monitoring method according to an embodiment of the present application;

[0029] Figure 4 This is a schematic diagram of an echo of a specific embodiment of the present application;

[0030] Figure 5 This is the second flow chart of the airport panoramic monitoring method according to an embodiment of the present application;

[0031] Figure 6 This is the third flow chart of the airport panoramic monitoring method according to an embodiment of the present application;

[0032] Figure 7 This is the fourth flow chart of the airport panoramic monitoring method according to an embodiment of the present application;

[0033] In the figure: 110, radar; 120, radar server; 130, camera device; 140, panoramic video surveillance server; 150, central control server; 160, user terminal; 200, airport panoramic monitoring system; 210, radar module; 220, central control module; 230, panoramic video surveillance module.

[0034] The realization of the objectives, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0035] To make the objectives, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments of this application, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this application.

[0036] In modern society, with the rapid development of the aviation industry, the importance of airports has become increasingly prominent. As a key transportation hub connecting cities, airport safety and operational efficiency are of paramount importance. To ensure the effective implementation of airport flight area safety management and monitoring, airport panoramic video surveillance systems have become an indispensable solution.

[0037] At present, the existing airport panoramic video surveillance system mainly collects airport images, sounds, temperature and other data in real time by installing cameras or sensors at various locations in the airport. Then, by processing and analyzing this data, it provides airport control personnel with complete airport information so that airport control personnel can make decisions and management.

[0038] However, in rainy, foggy and other weather conditions, the camera's line of sight may be obstructed by, for example, haze, raindrops, sunlight, etc. These factors will affect the camera's image acquisition process, causing the captured image quality to deteriorate. In addition, rainy, foggy and other weather conditions will also limit the visible range of the panoramic video surveillance system, making it impossible for airport control personnel to obtain accurate and comprehensive airport image information.

[0039] Radar is a radio device used to detect and observe targets, capable of operating in all weather conditions. Radar uses radio waves or microwaves as a transmission medium, transmitting pulses toward a target and then receiving reflected waves to estimate its direction, distance, and motion. Radar systems are widely used in weather forecasting, aircraft navigation, ship navigation, satellite navigation, military reconnaissance, and other fields.

[0040] To this end, the embodiments of the present application provide an airport panoramic monitoring system and an airport panoramic monitoring method, which combine a radar system and a panoramic video monitoring system. In rainy, foggy and other weather conditions or when visibility is very low, the radar system is used to detect flying objects at the airport and obtain accurate location information of the flying objects. The panoramic video monitoring system is used to obtain all-round and clear video images of the airport in real time, and the information of the flying objects monitored by the radar system is visualized and marked in the video image, so as to achieve more accurate and comprehensive monitoring of the airport, thereby improving the work efficiency of the airport tower staff and the flight safety of the airport aircraft.

[0041] For ease of understanding, this specification provides an example scenario of an airport panoramic monitoring system. Figure 1 The illustrated application environment includes several radars 110, a radar server 120, several cameras 130, a panoramic video surveillance server 140, a central control server 150, and user terminals 160. Each radar 110 can establish a communication connection with the radar server 120 via wired or wireless means; each camera 130 can also establish a communication connection with the panoramic video surveillance server 140 via wired or wireless means; and the radar server 120, the panoramic video surveillance server 140, the central control server 150, and the user terminals 160 can establish a communication connection via the internet or other means.

[0042] In this example scenario, all radars 110 can be distributed and installed at various locations throughout the airport, enabling them to scan and cover all airside aprons, runways, taxiways, and other areas requiring monitoring. Furthermore, users can configure the scanning parameters for each radar 110 on radar server 120 via user terminal 160. These parameters include (but are not limited to) scanning range, scanning frequency, and scanning sensitivity. Specifically, the scanning range of radar 110 must cover all possible flying objects around the airport, and the scanning frequency and scanning sensitivity must be set to appropriate values ​​to ensure that radar 110 can effectively detect flying objects in all weather conditions.

[0043] Likewise, all the camera devices 130 may be evenly distributed and installed at various locations in the airport, so that the camera devices 130 can monitor and cover all aprons, runways, taxiways and other areas that need to be monitored on the airside of the airport.

[0044] When monitoring an airport, all radars 110 begin scanning simultaneously, capturing the location information of all large objects and small targets. This location information is known as radar data, which each radar 110 transmits in real time to the radar server 120. Since airport monitoring primarily focuses on large objects within the airport, the radar server 120 processes this radar data (e.g., denoising) to retain the location information of large objects (i.e., target objects) and remove the location information of small targets.

[0045] Furthermore, the radar server 120 can send the processed position data of all target objects to the central control server 150; the central control server 150 can first convert the position data of each target object into data that can be recognized by the panoramic video surveillance server 140, and then synchronously transmit the converted position data of each target object to the panoramic video surveillance server 140.

[0046] In this scenario example, when all radars 110 start scanning, all cameras 130 can also start shooting the airport synchronously. The video images collected by each camera 130 are sent to the panoramic video surveillance server 140 in real time; the panoramic video surveillance server 140 stitches these video images to obtain a panoramic video image of the airport.

[0047] After receiving the converted position data of each target object, the panoramic video surveillance server 140 can mark the position data of the target object detected by the radar 110 in the panoramic video image according to the converted position data of each target object, or directly mark the target object in the panoramic video image in the form of a graphic.

[0048] In this example scenario, after obtaining the annotated panoramic video, panoramic video surveillance server 140 can transmit the annotated panoramic video to user terminal 160, which can display the annotated panoramic video to the user on a display. In this example scenario, user terminal 160 can include (but is not limited to): mobile terminals such as mobile phones, laptops, and PADs (tablet computers), as well as fixed terminals such as digital TVs and desktop computers. User terminal 160 is not specifically limited here.

[0049] Therefore, when tower monitoring personnel cannot see the flight status of aircraft or flying objects with the naked eye in rainy weather, they can clearly see the relevant information of flying objects through user terminal 160, effectively improving the efficiency and safety of airport operations.

[0050] With reference to the scenario example of the airport panoramic monitoring system in the aforementioned embodiment, the airport panoramic monitoring system in the embodiment of the present application is described in detail below.

[0051] Figure 2 This is a schematic diagram of the structure of the airport panoramic monitoring system according to the embodiment of the present application. Figure 2 As shown, the airport panoramic surveillance system 200 includes a radar module 210, a panoramic video surveillance module 230, and a central control module 220. In this embodiment, the central control module 220 establishes communication connections with the radar module 210 and the panoramic video surveillance module 230, respectively, and the radar module 210 and the panoramic video surveillance module 230 exchange data through the central control module 220. In some embodiments, the radar module 210 may also establish a direct communication connection with the panoramic video surveillance module 230, and the radar module 210 and the panoramic video surveillance module 230 may directly exchange data.

[0052] Among them, the radar module 210 is used to obtain radar data of the current airport, and process the radar data to obtain the position data of each target object in the airport, and send the position data of each target object to the central control module 220; the central control module 220 is used to convert the position data of each target object, and synchronously output the converted position data of each target object to the panoramic video monitoring module 230; the panoramic video monitoring module 230 is used to obtain the video image of the current airport, and mark each target object in the video image according to the converted position data of each target object.

[0053] In this embodiment, the radar module 210 may include multiple radars and a radar server capable of data processing. The radar module 210 can scan the airport in real time during rainy weather to capture the location information of all large objects, including aircraft, and the location information of small targets. This location information is the radar data mentioned above. The small targets mentioned here may include raindrops, birds, insects, etc. In the airport monitoring scenario, the main monitored targets are large objects in the airport, especially aircraft, vehicles, etc. These small targets will cause redundant data interference, thereby affecting the accuracy of the radar module 210's detection of flying objects, etc. Therefore, it is necessary to perform certain data processing on the radar data to remove the location information of these small targets and retain the location information of large objects that need to be monitored. Based on this, the target objects in this embodiment can include any object that needs to be monitored, such as aircraft and vehicles.

[0054] Specifically, radar transmits radio waves and receives signals reflected from objects to determine an object's position, distance, and speed. Before the radar scans an airport, it can first set its scanning parameters based on actual weather conditions and monitoring requirements. Scanning parameters can include sensitivity, scanning range, and scanning frequency. The scanning range needs to cover all possible flying objects around the airport. In addition, in rainy, foggy, and other weather conditions, especially rain, the radar's sensitivity needs to be set higher because the radio waves emitted by the radar may be scattered by raindrops. However, the target object is much larger than a raindrop, so the radar can still accurately detect the target object.

[0055] After the radar scans the current airport, it generates radar data. The radar then transmits this data to a radar server, which filters the data to retain the location data of each target object. For example, the radar data can be denoised to eliminate interference from irrelevant objects such as raindrops and birds.

[0056] After the radar module 210 obtains the position data of each target object, it needs to transmit this data to the panoramic video monitoring module 230 in real time and synchronously; in addition, since the data formats of the radar module 210 and the panoramic video monitoring module 230 are different, the data format transmitted by the radar module 210 to the panoramic video monitoring module 230 needs to be converted into a data format that the panoramic video monitoring module 230 can recognize.

[0057] In this embodiment, the central control module 220 converts the position data of each target object acquired by the radar module 210 into a data format recognizable by the panoramic video monitoring module 230. Specifically, the central control module 220 may include a central control server, which includes a data processing unit (e.g., a CPU or other data processing chip) that converts the position data of each target object.

[0058] It should be noted that the data conversion mentioned here mainly includes coordinate data conversion and data format conversion. As we all know, the radar data obtained by radar scanning is mainly the polar coordinate data (i.e., distance and angle) of the target object relative to the radar, and the panoramic video monitoring module 230 needs to use the Cartesian coordinate data of the target object. Therefore, the position data of each target object needs to be converted into coordinate data; in addition, in order to unify the data format into the data format corresponding to the panoramic video monitoring module 230, the position data of each target object also needs to be converted into data format.

[0059] The central control module 220 converts the position data of each target object and then synchronously transmits the converted position data of each target object to the panoramic video monitoring module 230 in real time.

[0060] In this embodiment, the panoramic video surveillance module 230 may include multiple cameras and a panoramic video surveillance server. The cameras may be evenly distributed and installed at various locations throughout the airport, enabling them to monitor and cover all airside aprons, runways, taxiways, and other areas requiring monitoring. Each camera captures the airport in real time, and each camera sends the captured video images to the panoramic video surveillance server. The panoramic video surveillance server may include a built-in data processing unit that processes the image data. After receiving the video images from each camera, the panoramic video surveillance server may stitch all the video images together to produce a panoramic video image of the airport.

[0061] It should be noted that each camera device needs to collect data simultaneously with each radar to ensure that the radar data obtained by the radar module 210 is subsequently synchronized with the video image information obtained by the panoramic video monitoring module 230.

[0062] After obtaining the panoramic video image, the panoramic video monitoring module 230 may pre-process the panoramic video image. Alternatively, after receiving the video images sent by each camera device, the module may pre-process each video image and then stitch the processed video images together to obtain the panoramic video image. In this embodiment, the pre-processing of the video image may include noise removal, brightness level adjustment, and image enhancement, so as to facilitate subsequent display.

[0063] Furthermore, after receiving the converted position data of each target object from the central control module 220, the panoramic video monitoring module 230 marks each target object in the panoramic video image based on the position data of each target object. In this embodiment, the marking form can be graphic, text, or a combination of graphic and text, and the marking form is not specifically limited here.

[0064] For example, if the target object is an airplane, you can directly mark the airplane in the panoramic video image with a dot, a label containing information about the target object, or a combination of a dot and a label containing information. It should be noted that the label can include relevant information such as the aircraft's altitude and flight number.

[0065] Thus, in rainy, foggy, and other weather conditions, radar module 210 can detect all flying objects at the airport, and the radar data detected by the radar can be combined with the video image data obtained by panoramic video monitoring module 230, effectively resolving the problems of low visibility, incomplete, and inaccurate monitoring by panoramic video monitoring module 230 in such weather conditions. In addition, by visually annotating the flying object position information obtained by radar in the panoramic video image, the overall visual effect of the panoramic video image is improved. Tower personnel can clearly and comprehensively understand the current situation at the airport through the annotated panoramic video image, which is more conducive to airport management and aircraft command, improving work efficiency, and helping to avoid collisions and aircraft getting lost during takeoff and landing, greatly improving flight safety.

[0066] Rain, snow, and clouds in the atmosphere attenuate radar waves and generate echoes, severely impacting radar observation of target objects. Furthermore, the water content of rain and snow clouds is directly proportional to the intensity of their echoes. For example, radar echoes from rain and snow clouds with low water content appear as loose, cotton-like bright spots with no distinct edges in radar images, while radar echoes from rain and snow clouds with high water content appear as loosely bounded, continuous patches of strong interference.

[0067] In order to suppress the interference effects of rain, snow, and heavy fog on the radar data obtained by the radar module 210, in some embodiments, the radar module 210 can be specifically used to: convert radar data into a radar image, and obtain the grayscale value change rate of each pixel point in the radar image; when the grayscale value change rate of the pixel point is less than a preset threshold, determine the pixel point as a noise point; and denoise the noise point according to the sign of the grayscale value change rate of the noise point.

[0068] It should be noted that radar images are a form of representation of radar data. Radar images are two-dimensional or three-dimensional images created based on radar data. They can help users understand radar data more intuitively.

[0069] It can be understood that each pixel in the radar image corresponds to a radar data point. If the radar data contains the location information of a large object (i.e., the target object), the radar data will appear in the radar image as follows: the grayscale value of the corresponding pixel point is significantly different from the grayscale values ​​of the surrounding points. If the radar data point corresponds to the location information of rain and snow noise, the radar data will appear in the radar image as follows: the grayscale value of the corresponding pixel point is slightly different from the grayscale values ​​of the surrounding points. This is because the radar signal reflected by rain and snow noise is usually weak, so it appears in the radar image as a small change in the grayscale value of the pixel point.

[0070] Based on this, a preset threshold can be set. This threshold serves as the critical value for detecting the grayscale value change rate of target objects in radar images. This threshold can be used to distinguish target objects from rain and snow noise in radar images. It should be noted that the preset threshold can be manually set to an appropriate value through extensive experimentation and is not specifically limited here.

[0071] Specifically, the grayscale change rate of each pixel in the radar image can be determined. This grayscale change rate can be determined by comparing the grayscale value of the pixel with the grayscale values ​​of surrounding pixels. For details, reference can be made to existing methods for determining grayscale change rates, which will not be elaborated here. Furthermore, the grayscale change rate of each pixel in the radar image is compared with a preset threshold. Pixels with a grayscale change rate greater than or equal to the preset threshold are kept unchanged in grayscale; pixels with a grayscale change rate less than the preset threshold are determined to be noise points, which are rain or snow noise.

[0072] After identifying the noise points in the radar image, we need to further determine the sign of the grayscale value change rate of the noise point to determine the denoising method for that noise point. It should be noted that the sign of the grayscale value change rate is simply the positive or negative sign of the grayscale value change rate. It represents the direction of the grayscale value change and can be used to determine whether the area is the leading edge or the trailing edge. For example, when the sign of the grayscale value change rate is negative, it indicates that the area is the trailing edge, indicating that the object has gone from presence to absence, so the grayscale value change rate is decreasing; when the sign of the grayscale value change rate is positive, it indicates that the area is the leading edge, indicating that the object has gone from absence to presence, so the grayscale value change rate is increasing. Based on this property, different processing methods can be used to deal with rain and snow noise on the leading edge and rain and snow noise on the trailing edge respectively.

[0073] One approach to handling noise points with a positive grayscale change rate (i.e., leading edge) is to retain the grayscale of the noise point, as changes at this location may also represent the actual target object. Specifically, only the grayscale of pixels with a grayscale change rate above a certain threshold—that is, only those with relatively rapid grayscale changes—can be retained to reduce the impact of rain and snow noise.

[0074] One approach to dealing with noise points with a negative grayscale change rate (i.e., trailing edges) is to remove rain and snow noise along the trailing edge to avoid misjudgments. Specifically, this can be done by removing only pixels with a grayscale change rate below a certain threshold—that is, pixels with relatively slow grayscale changes.

[0075] Thus, the above-described method can effectively eliminate the interference of rain and snow noise on radar data, retain the position data of each target object, and obtain more accurate target object position data. The radar module 210 may also process radar data by other processing steps. For details, please refer to the data processing steps in the embodiment of the airport panoramic monitoring method described below, which will not be described in detail here.

[0076] It is understandable that the position data of the target object acquired by the radar module 210 is generally polar coordinate data (ie, distance and angle) of the object relative to the radar, and therefore the position data of the target object includes the polar coordinate data of the target object.

[0077] In some embodiments, the central control module 220 may include middleware, and the central control module 220 can be specifically used to: perform coordinate conversion on the polar coordinate data of each target object to obtain the intermediate coordinate data of each target object in the coordinate system of the panoramic video monitoring module 230; use the middleware to convert the data format of the intermediate coordinate data of each target object into a target data format, and the target data format is the data format corresponding to the panoramic video monitoring module 230; and synchronously transmit the intermediate coordinate data of each target object after the data format conversion to the panoramic video monitoring module 230.

[0078] It should be noted that the intermediate coordinate data can be the coordinate data of the target object in the Cartesian coordinate system. Because the panoramic video monitoring module 230 usually uses the Cartesian coordinates of the object, it is necessary to convert the polar coordinate data of the target object into Cartesian coordinate data. The main purpose of doing this is to convert the data into more intuitive and easier to process Cartesian coordinate data, and to facilitate synchronous display in the panoramic video monitoring module 230, positioning the target object identified by the radar in the panoramic video monitoring module 230, and reflecting the real-time status of the target object. The specific conversion method of the central control module 220 can refer to the coordinate conversion process in the following embodiment of the airport panoramic monitoring method, which will not be introduced in detail here.

[0079] In addition, in order to communicate between different systems, a middleware is usually required to parse the data. The middleware can parse the data formats of different systems and perform necessary data conversion. Therefore, after converting the polar coordinate data of each target object into the intermediate coordinate data of each target object, the middleware in the central control module 220 can be used to convert the data format of each intermediate coordinate data into the data format corresponding to the panoramic video monitoring module 230. For example, if the data output by the radar module 210 is in XML format, and the panoramic video monitoring module 230 only receives data in JSON format, then the middleware can convert the XML format data output by the radar module 210 into JSON format, and then transmit the JSON format data to the panoramic video monitoring module 230. In this way, the data format is unified, which facilitates the subsequent data processing by the panoramic video monitoring module 230.

[0080] The processing process of the target object's position data and the data synchronization transmission process by the central control module 220 can refer to the process in the following airport panoramic monitoring method embodiment, which will not be described in detail here.

[0081] In some embodiments, the panoramic video surveillance module 230 can be specifically used to: determine the geographic coordinate data of each target object based on the intermediate coordinate data of each target object; perform coordinate conversion on the geographic coordinate data of each target object to obtain the plane coordinate data of each target object, where the plane coordinate data of the target object is the position coordinate of the target object in the coordinate system of the video image; and mark each target object in the video image based on the plane coordinate data of each target object.

[0082] Specifically, after receiving the intermediate coordinate data of each target object sent by the central control module 220, the panoramic video monitoring module 230 can first convert the intermediate coordinate data of each target object into the geographic coordinate data of each target object. It can be understood that the position data of the target object obtained by the radar module 210 is the polar coordinate data of the target object relative to the radar. This is a type of coordinate data related to the position of the radar device and has nothing to do with the geographic location of the target object itself. Therefore, it is necessary to convert the intermediate coordinate data of each target object into the geographic coordinate data of each target object. The specific coordinate conversion process can be referred to the coordinate conversion process in the following method embodiment, and will not be described in detail here.

[0083] Furthermore, after obtaining the geographic coordinate data of each target object, it is necessary to perform coordinate conversion on the geographic coordinate data of each target object to obtain the plane coordinate data of each target object. It is understandable that in order to match the geographic coordinate data of the target object with the panoramic video image coordinate data obtained by the panoramic video monitoring module 230 to mark the target object in the panoramic video image, it is necessary to first convert the geographic coordinate data into plane coordinate data. It should be noted here that the physical meanings represented by geographic coordinate data and plane coordinate data are different. Geographic coordinate data describes the position of the target object on the surface of the earth, while plane coordinate data describes the position of the target object on the image array. The purpose of this step is to convert the position information of the target object in the geographic coordinate system into position information in the image coordinate system, so as to accurately mark the actual position of the target object scanned by the radar on a certain frame of the panoramic video image. The specific coordinate conversion process can refer to the coordinate conversion process in the following method embodiment, which will not be described in detail here.

[0084] In some embodiments, the radar module 210 can also be specifically used to: obtain basic information of each target object and send the basic information of each target object to the central control module 220; the central control module 220 can also be specifically used to: synchronously send the basic information of each target object to the panoramic video monitoring module 230; the panoramic video monitoring module 230 can also be specifically used to: use the circle function and according to the plane coordinate data of each target object, perform graphic annotation on each target object in the video image, and use the putText function and according to the basic information of each target object, perform text annotation on each target object in the video image.

[0085] In this embodiment, a method of visually marking each target object on the panoramic video image based on the plane coordinate data of each target object mainly uses a method of combining graphic marking and text marking.

[0086] Specifically, after detecting each target object, if the target object is an aircraft, radar module 210 can establish a data link with the ATC (Air Traffic Control) system based on the flight plan to obtain basic information such as the target object's flight number and estimated landing time. Before an aircraft takes off, the flight plan is submitted to the relevant authorities and includes information such as the aircraft's flight number, takeoff time, and estimated flight path. Radar module 210 compares real-time detection data with the flight plan to determine which flight plan matches the target object currently detected by the radar. It then retrieves the matching flight plan to obtain the basic information about the target object in the flight plan.

[0087] Furthermore, aircraft are typically equipped with a transponder, which automatically responds to radar signals by transmitting a signal containing the aircraft's identity, real-time flight parameter codes, and other data. Upon receiving this signal, radar module 210 can directly obtain basic information about the aircraft. This type of information exchange using a transponder is known as a secondary radar or SSR (Secondary Surveillance Radar) system.

[0088] After the radar module 210 acquires the basic information of each target object through any of the aforementioned methods, it is also synchronously transmitted to the panoramic video monitoring module 230 through the central control module 220. It should be noted that the basic information of each target object can be sent to the panoramic video monitoring module 230 separately through the central control module 220, or the basic information of each target object and its location data can be sent to the central control module 220 together, and then processed by the central control module 220 before being synchronously transmitted to the panoramic video monitoring module 230.

[0089] Panoramic video surveillance module 230 receives basic information about each target object from central control module 220. When annotating each target object, panoramic video surveillance module 230 can utilize OpenCV algorithms. Specifically, the circle function can be used to annotate each target object graphically within the video image based on its planar coordinate data. The putText function can be used to annotate each target object textually within the video image based on its basic information. The specific annotation process can be referenced in the method embodiments described below and will not be described in detail here.

[0090] In some embodiments, the radar module 210 includes at least one radar, and the panoramic video monitoring module 230 includes at least one camera device; each radar is used to obtain radar data of the current airport; each camera device is used to obtain video images of the current airport; the central control module 220 can also be used to align the time starting point and synchronize the data frames of each radar and each camera device.

[0091] It is understandable that in a system with multiple data acquisition devices, the data rates between different devices may vary, and the collected information is often based on different coordinate systems, and even the types of coordinates are different. For example, the data collected by the radar and camera devices in this embodiment is data in different coordinate systems. Therefore, before subsequent data processing, this data needs to be fused. Before data fusion, the radar and camera devices need to be calibrated. Calibration mainly involves timeline calibration. The purpose of timeline calibration is to align the time starting points of the data and to correspond different types of data frames.

[0092] Based on this, it is necessary to align the time starting points and synchronize the data frames of all radars and all cameras to achieve the calibration on the timeline mentioned above. In this embodiment, the method for achieving the time starting point alignment of the radar and the camera device can be: setting a working delay for the radar, or setting a working delay for the camera, so that the time starting points of the radar and the camera are the same. In addition, because the refresh rate of the radar output data is different from the frame rate of the camera, it is necessary to synchronize the data frames of the two so that the data used for fusion of the two devices describes a moment as close as possible. The specific time starting point alignment and data frame synchronization process can refer to the time starting point alignment and data frame synchronization process in the following method embodiment, which will not be described in detail here.

[0093] In some embodiments, the panoramic video monitoring module 230 may further include a display, which is used to display the video image with graphic annotations and text annotations.

[0094] Specifically, the panoramic video surveillance module 230 may include a user terminal with a display capable of displaying the annotated video image. The display process typically includes two sub-steps: image rendering and image playback. Furthermore, the display may also display a user interface, where the user can control the playback of the annotated video image by operating controls within the user interface. The specific display process can be referenced in the method embodiments described below and will not be described in detail here.

[0095] Based on the above embodiments, the present application also provides a method for panoramic airport monitoring, which can be applied to the panoramic airport monitoring system described in any of the above embodiments. The following uses the method executed by the panoramic airport monitoring system as an example to illustrate the method.

[0096] Figure 3 This is one of the flow charts of the airport panoramic monitoring method according to the embodiment of the present application. Figure 3As shown, the airport panoramic monitoring method may include the following steps:

[0097] Step 310: The radar module obtains radar data of the current airport, and the panoramic video surveillance module obtains video images of the current airport.

[0098] Step 320: The radar module processes the radar data to obtain the position data of each target object in the airport, and sends the position data of each target object to the central control module.

[0099] Step 330: The central control module performs data conversion on the position data of each target object, and synchronously outputs the converted position data of each target object to the panoramic video monitoring module.

[0100] Step 340: The panoramic video monitoring module marks each target object in the video image according to the converted position data of each target object.

[0101] In this embodiment, the radar module includes multiple radars, and the panoramic video surveillance module includes multiple cameras. The radars can detect the current airport and generate corresponding radar data. Radar data can include the target object's position, distance, and speed in a polar coordinate system with the radar as the reference point. The cameras can capture the current airport and generate corresponding video images. The panoramic video surveillance module receives the video images output by each camera in real time and stitches them together using stitching technology to generate a panoramic video image of the airport.

[0102] However, before the radar data is collected by the radar and the camera device collects video images, it is necessary to initialize the radar module and the panoramic video monitoring module, and jointly calibrate all radars and all camera devices.

[0103] Initializing the radar modules involves turning them on and setting appropriate scanning parameters for each radar. These parameters include scanning range, frequency, and sensitivity. The scanning range should cover all possible flying objects around the airport, while the frequency and sensitivity should be set appropriately for more effective detection.

[0104] As an example, in rainy weather, the radar's scanning range can be 5 kilometers around the airport, the scanning frequency can be 10 scans per second, and the sensitivity is set to be able to detect objects the size of a vehicle.

[0105] Initializing the panoramic video surveillance module involves setting the shooting range and angle for each camera. For example, each monitoring point in the panoramic video surveillance module can have 4-12 cameras, each with a 72° viewing angle. This allows for coverage of the entire airport apron and runway through one or more panoramic viewpoints.

[0106] Understandably, because radar and cameras are separate devices, their data rates can vary, and the information they collect is often based on different coordinate systems, even with different coordinate types. Typically, the operating frequencies and timings of the radar and panoramic video surveillance modules may differ, necessitating calibration and synchronization to ensure that radar and panoramic video surveillance data are correctly aligned during subsequent data processing. Only when the timelines of the two modules are aligned and synchronized can subsequent data acquisition and processing be accurate and efficient.

[0107] For example, the position data of a target object collected by radar is expressed in polar coordinates relative to the radar; whereas, the image data collected by a camera is primarily in a Cartesian coordinate system. Therefore, the polar coordinate data of the target object collected by the radar must be converted into Cartesian coordinates and then fused with the image data. This data fusion process ensures that both data describe airport-related data at the same time as closely as possible. Therefore, prior to data collection, all radars and cameras must be calibrated together.

[0108] The following is a detailed introduction to the timeline calibration process of radar and camera devices.

[0109] First of all, it should be noted that the calibration of the radar and camera devices on the timeline mainly includes the alignment of the time starting point and the synchronization of the data frame. In addition, the central control module can perform the calibration process of the radar and camera devices on the timeline, that is, the central control module collects and integrates the relevant data of all radars and camera devices, and is responsible for ensuring the time alignment and data frame synchronization of radar data and panoramic video surveillance data.

[0110] For the process of aligning the time starting points of the radar and camera, you can set a suitable delay for the camera, or set a suitable delay for the radar. Taking setting a suitable delay for the camera as an example, the time coordinates of the radar and the time coordinates of the camera satisfy the relationship represented by the following formula:

[0111]

[0112] Where, is the time coordinate of the radar, is the time coordinate of the camera device, It should be noted that the delay can be set manually based on the hardware characteristics of the radar and camera devices. For example, it can be set based on the data processing speed and data transmission speed of the radar and camera devices respectively. In addition, signal propagation delay needs to be considered based on the physical device installation distance of the radar and camera devices and the medium propagation speed.

[0113] Specifically, after determining the time delay of the camera device, the time delay can be set in advance. Before the radar or camera device starts to collect data, the central control module first sends a synchronization signal to them, allowing the radar or camera device to start collecting data the moment the signal is received. Since the time delay of the camera device has been set, the radar and camera device will start working at the same time, thereby achieving synchronization of the time starting point and preventing data deviation caused by time asynchrony.

[0114] As an example, if the radar takes t1 time to start working after receiving the synchronization signal, and the camera takes t2 time to start working after receiving the synchronization signal, then as long as the central control module sends synchronization signals to these two devices at the same time and sets the delay of the camera to |t1-t2|, it can ensure that the radar and camera start collecting data at the same time.

[0115] To synchronize the data frames output by the radar and camera, the following relationships can be established:

[0116]

[0117] Where, and represent functions of taking absolute value and smaller value respectively, Indicates the data frame number of the radar output data, Indicates the data frame number of the camera output data, Indicates the period during which the radar sends data. Indicates the period during which the camera sends data. 、 and radar data rate , video stream frame rate There is an inverse relationship between them.

[0118] In actual application, the radar data transmission cycle can use the fast and slow time dimension joint estimation algorithm to estimate the target parameters. The number of pulses used for one estimation is 128, and the single pulse period is 768us. Since the acquisition and processing are carried out simultaneously, the radar data refresh cycle is still 128x0.768=98ms (milliseconds), which is the radar data transmission cycle. Can be 98ms; video stream frame rate 30 frames per second.

[0119] It should also be noted that data frame sequence numbers are typically automatically generated by the data acquisition device (i.e., the radar and camera in this embodiment) during the data acquisition process and output along with the data. The data frame sequence number is typically an increasing value that identifies the order of each frame of data. Therefore, even before the radar and camera begin data acquisition, each frame of data will have a corresponding sequence number once data acquisition begins. At this stage, the device configuration is typically set to start with sequence number 0 or 1. After data acquisition, the sequence number can be used to identify the acquisition time of each frame in the data set, thereby facilitating subsequent time alignment and data frame synchronization.

[0120] After the radar and camera device are calibrated on the timeline in the above manner, the radar and camera device can be controlled to start data collection.

[0121] The following is a detailed description of the process in step 310 where the radar module acquires radar data of the airport and the panoramic video monitoring module acquires video images of the airport.

[0122] First, all radars in the radar module transmit radio waves and receive signals reflected from different objects to obtain radar data such as the direction, distance, and speed of different objects. It is important to note that in rainy weather, especially rain, the radio waves emitted by the radar may be scattered by raindrops. However, since the volume of the target object is much larger than the raindrop, the radar can still accurately detect the target object as long as the sensitivity setting is high enough.

[0123] After acquiring radar data, the radar transmits it to the radar server in the radar module, which processes the radar data. In step 320, the radar data processing primarily involves denoising and anti-interference processing, thereby eliminating interference from irrelevant objects such as raindrops and birds that could interfere with the target object's position data.

[0124] Specifically, a filter can be used to perform preliminary denoising on the radar data. In this embodiment, a moving average filter is used to perform denoising on the radar data. The moving average filter takes the data of the current point and the data of the previous M-1 points, calculates their average value, and uses the average value as the output value of the current point. Its mathematical formula can be expressed as:

[0125]

[0126] in, is the radar data at the nth moment, M is the window length of the moving average filter, The radar data output by the moving average filter at time n. In this way, the impact of sudden changes (i.e., noise) on the output will be dispersed to multiple output points, thereby reducing the impact of noise.

[0127] Furthermore, because radars operating in adjacent areas on the same frequency band receive not only their own pulse signals, but also pulse signals from other radars operating in the same frequency band, this can cause co-frequency interference on the radar data. Therefore, in this embodiment, co-frequency interference removal is also required on the radar data.

[0128] For the same target object, echoes reflected from the target object should appear at the same distance unit on adjacent scan lines emitted by the radar. However, co-channel interference clutter does not have this correlation. Therefore, echo correlation technology can be used to suppress co-channel interference. Specifically, two or more adjacent scan lines can be detected. Taking the suppression of co-channel interference on two scan lines as an example, if the same or similar echoes exist at the same distance on two adjacent scan lines, they are considered to be echoes reflected from the target object and are retained. Otherwise, they are considered to be echo interference and are removed.

[0129] Figure 4 This is a schematic diagram of an echo of a specific embodiment of the present application. Figure 4 As shown in the figure, as an example, when scan lines 1 and n both have echoes at the same distance, the output retains the echo. When only scan line 1 has an echo at a certain distance, or only scan line n has an echo at a certain distance, the output removes these echoes. It should be noted that the more scan lines involved in the correlation operation, the better the co-channel interference suppression effect.

[0130] Figure 5 This is the second flowchart of the airport panoramic monitoring method according to an embodiment of the present application.

[0131] Furthermore, rain, snow, and clouds in the atmosphere can attenuate radar waves and generate echoes, which can seriously affect the radar's observation of target objects. In order to suppress the interference of rain, snow, and fog on the radar module's acquisition of radar data, Figure 5 As shown, in some embodiments, the radar module processes the radar data in step 320 to obtain the position data of each target object in the airport, and the following steps may also be included:

[0132] Step 510: Convert the radar data into a radar image, and obtain the grayscale value change rate of each pixel in the radar image.

[0133] Step 520: When the grayscale value change rate of a pixel point is less than a preset threshold, the pixel point is determined to be a noise point.

[0134] Step 530: De-noising the noise points according to the signs of the grayscale value change rates of the noise points.

[0135] It should be noted that radar images are a form of radar data. Radar images are two-dimensional or three-dimensional images created based on radar data, which can help users more intuitively understand radar data. Furthermore, the preset threshold is the critical value for detecting the rate of change in the grayscale value of target objects in radar images. This value can be manually determined through extensive experimentation, and no specific limit is imposed here.

[0136] It can be understood that each pixel in the radar image corresponds to a radar data point. If the radar data contains the location information of a large object (i.e., the target object), the radar data will appear in the radar image as follows: the grayscale value of the corresponding pixel point is significantly different from the grayscale values ​​of the surrounding points. If the radar data point corresponds to the location information of rain and snow noise, the radar data will appear in the radar image as follows: the grayscale value of the corresponding pixel point is slightly different from the grayscale values ​​of the surrounding points. This is because the radar signal reflected by rain and snow noise is usually weak, so it appears in the radar image as a small change in the grayscale value of the pixel point.

[0137] Specifically, the grayscale change rate of each pixel in the radar image can be determined. This grayscale change rate can be determined by comparing the grayscale value of the pixel with the grayscale values ​​of surrounding pixels. For details, reference can be made to existing methods for determining grayscale change rates, which will not be elaborated here. Furthermore, the grayscale change rate of each pixel in the radar image is compared with a preset threshold. Pixels with a grayscale change rate greater than or equal to the preset threshold are kept unchanged in grayscale; pixels with a grayscale change rate less than the preset threshold are determined to be noise points, which are rain or snow noise.

[0138] After identifying the noise points in the radar image, we need to further determine the sign of the grayscale value change rate of the noise point to determine the denoising method for that noise point. It should be noted that the sign of the grayscale value change rate is simply the positive or negative sign of the grayscale value change rate. It represents the direction of the grayscale value change and can be used to determine whether the area is the leading edge or the trailing edge. For example, when the sign of the grayscale value change rate is negative, it indicates that the area is the trailing edge, indicating that the object has gone from presence to absence, so the grayscale value change rate is decreasing; when the sign of the grayscale value change rate is positive, it indicates that the area is the leading edge, indicating that the object has gone from absence to presence, so the grayscale value change rate is increasing. Based on this property, different processing methods can be used to deal with rain and snow noise on the leading edge and rain and snow noise on the trailing edge respectively.

[0139] One approach to handling noise points with a positive grayscale change rate (i.e., leading edge) is to retain the grayscale of the noise point, as changes at this location may also represent the actual target object. Specifically, only the grayscale of pixels with a grayscale change rate above a certain threshold—that is, only those with relatively rapid grayscale changes—can be retained to reduce the impact of rain and snow noise.

[0140] One approach to dealing with noise points with a negative grayscale change rate (i.e., trailing edges) is to remove rain and snow noise along the trailing edge to avoid misjudgments. Specifically, this can be done by removing only pixels with a grayscale change rate below a certain threshold—that is, pixels with relatively slow grayscale changes.

[0141] Furthermore, the SPxFTCPro function can be combined with the SPxRunProcess function to adjust the radar equipment's technical parameters, making the radar echo appear clearer on the screen, thereby further suppressing rain and snow interference. The SPxFTCPro function automatically detects the leading edge of a target object (such as an aircraft) from the input radar data, identifies and removes interfering information such as rain and snow noise. The SPxRunProcess function further processes the data processed by the SPxFTCPro function, outputting the processed radar data in a format suitable for the panoramic video surveillance module. This approach effectively eliminates the interference of rain and snow noise on radar data, preserves the position data of each target object, and obtains more accurate target location data.

[0142] In some implementations, the Laplacian operator may be further used to perform edge detection on the radar echo image, and then a median filter algorithm may be used to effectively remove ray-shaped noise in the radar echo image.

[0143] It should be noted that ray-shaped noise can originate from radar malfunctions or temporary external interference. This type of noise typically manifests as ray-shaped or strip-shaped noise in radar echo images. The median filter is an effective nonlinear filtering method primarily used to eliminate salt-and-pepper noise, i.e., black and white dots. Since ray-shaped noise typically appears as lines of extremely high or low brightness in radar echo images, the median filter is effective in eliminating this type of noise. The specific edge detection process and the use of the median filter to remove ray-shaped noise can be referenced in existing procedures and will not be elaborated here.

[0144] For example, if the radar module detects an aircraft taxiing on the runway at (120.63°E, 31.14°N), its position data is (120.63°E, 31.141°N) and its speed is 30 km / h, the program code for the raw radar data obtained by the radar module can be:

[0145] {

[0146] "plane1":{"x":2000,"y":3000,"z":0},

[0147] "plane2":{"x":2500,"y":3500,"z":0},

[0148] "car1":{"x":2600,"y":3700,"z":0},

[0149] "truck1":{"x":2700,"y":3800,"z":0},

[0150] }

[0151] Due to the interference caused by raindrops, birds and other irrelevant objects, the radar data is processed by the radar module to eliminate these noises and retain the information of the target object. The program code for the data obtained after processing and denoising the original radar data can be as follows:

[0152] {

[0153] "plane1":{"x":2000,"y":3000},

[0154] "plane2":{"x":2500,"y":3500}

[0155] }

[0156] These data represent the position data of the two aircraft. Therefore, after the radar module processes the radar data, the accurate position data of the target object can be obtained.

[0157] After the panoramic video monitoring module receives the video images sent by each camera device in real time, the panoramic video monitoring module can first pre-process these video images. In this embodiment, the pre-processing of the video images includes at least smoothing, histogram equalization, edge detection and image enhancement, so as to facilitate better display of the video images.

[0158] First, in order to eliminate noise in the video image, this embodiment uses a Gaussian filter to perform denoising on the video image. The mathematical formula is:

[0159]

[0160] In this formula, (x, y) is the position of the pixel in the video image, σ is the standard deviation of the Gaussian function, and σ describes the degree of smoothness. The larger the σ value, the more obvious the smoothing effect. G(x, y) is the position of the pixel in the video image output by the Gaussian filter.

[0161] In order to enhance the contrast of the video image and make the brightness and darkness of the video image uniform, this embodiment uses histogram equalization, and its mathematical formula is:

[0162]

[0163] In the above formula, is the probability distribution of a single pixel value r in the entire video image; is the frequency of occurrence of pixel value r, and N is the number of all pixels in the video image; is the cumulative distribution function.

[0164] In order to better display objects in the video image, in this embodiment, edge detection can be performed on the video image. Specifically, the Sobel operator can be used for this purpose. The calculation formula of the Sobel operator is:

[0165]

[0166] In these two 3x3 matrices, Gx represents the horizontal gradient, and Gy represents the vertical gradient. Convolving these two matrices with the video image's pixel matrix yields two new matrices. Taking the square root of the sum of these squares yields a video image containing edge information.

[0167] For the darker parts of the video image, its brightness can be enhanced. This can be achieved by using gamma correction. The corresponding mathematical formula is:

[0168]

[0169] In this formula, S is the grayscale value of the output (processed) video image, R is the grayscale value of the input (original) video image, c and γ are both constants, c is the magnification factor, and γ is the exponential value. The larger the γ value, the brighter the video image overall.

[0170] The above describes the process by which the radar module processes radar data to obtain the location data of each target object, and the panoramic video monitoring module processes the video image. After obtaining the location data of each target object, the radar module needs to synchronously transmit this location data to the panoramic video monitoring module so that the panoramic video monitoring module can perform subsequent operations.

[0171] The following describes in detail the process of synchronously transmitting the position data of each target object.

[0172] The data synchronization transmission between the radar module and the panoramic video surveillance module is mainly achieved through the central control module. Specifically, the radar module transmits the position data of each target object to the central control module in real time. The central control module needs to convert the position data of each target object and then synchronously transmit the converted position data of each target object to the panoramic video surveillance module.

[0173] It is understood that the radar module typically outputs the polar coordinate data (i.e., distance and angle) of the target object relative to the radar. Therefore, the target object position data described in this embodiment includes the target object's polar coordinate data. However, the panoramic video surveillance module typically uses the target object's Cartesian coordinates, so it is necessary to convert the target object's polar coordinate data into Cartesian coordinate data. In addition, to enable communication between different systems, middleware is typically required to parse the data. Therefore, after converting the polar coordinate data of each target object into intermediate coordinate data for each target object, the middleware in the central control module can be used to convert the data format of each intermediate coordinate data into the data format corresponding to the panoramic video surveillance module. This ensures a unified data format and facilitates subsequent data processing by the panoramic video surveillance module.

[0174] Figure 6 This is the third flowchart of the airport panoramic monitoring method according to an embodiment of the present application.

[0175] Based on this, in some embodiments, in step 330, the central control module converts the position data of each target object, which may include the following steps:

[0176] Step 610: performing coordinate conversion on the polar coordinate data of each target object to obtain the intermediate coordinate data of each target object in the coordinate system of the panoramic video monitoring module.

[0177] Step 620: Convert the data format of the intermediate coordinate data of each target object into a target data format, where the target data format is a data format corresponding to the panoramic video monitoring module.

[0178] Step 630: Synchronously transmit the intermediate coordinate data of each target object after data format conversion to the panoramic video monitoring module.

[0179] It should be noted that the coordinate system of the panoramic video monitoring module described in this embodiment is a Cartesian coordinate system.

[0180] Specifically, the polar coordinate data of the target object usually includes the distance l and angle θ of the target object relative to the radar (the angle is usually in degrees, but needs to be converted to radians during calculation). The polar coordinate data of each target object can be converted into Cartesian coordinate data using the following formula to obtain the intermediate coordinate data of each target object:

[0181]

[0182]

[0183] Where (X, Y) is the converted Cartesian coordinate of the target object, l is the distance of the target object relative to the radar, and θ is the angle of the target object relative to the radar. Furthermore, assuming the radar is located at the center of the panoramic image, the coordinates of the radar relative to the panoramic image are (X0, Y0). The position of the target object on the panoramic image (X', Y') can be calculated using the following formula:

[0184] X'=X0+X

[0185] Y'=Y0+Y

[0186] It should be noted that the starting value of angle θ is determined by the radar module's settings. For example, angle θ can be positive clockwise from due north or counterclockwise from due east. Before conversion, the direction of angle θ must be determined and converted to radians, where radians = degrees * π / 180.

[0187] In this way, the central control module can convert the target object position obtained by radar scanning from polar coordinates to Cartesian coordinates in the panoramic video monitoring module, and obtain the intermediate coordinate data (X', Y') of each target object. Thus, these polar coordinate data are converted into more intuitive and easier to process Cartesian coordinate data, which is conducive to subsequent data synchronization and other processing.

[0188] Furthermore, after obtaining the intermediate coordinate data of each target object, if the intermediate coordinate data of each target object is inconsistent with the data format of the panoramic video surveillance module, the central control module also needs to convert the data format of the intermediate coordinate data of each target object to ensure the uniformity of the data format, which is convenient for the panoramic video surveillance module to process the data subsequently.

[0189] As an example, if the radar module outputs data in XML format, and the panoramic video surveillance module only receives data in JSON format, the middleware can convert the XML format data output by the radar module into JSON format, and then transmit the JSON format data to the panoramic video surveillance module.

[0190] In this embodiment, since the position of the target object may change in real time, the process of synchronously transmitting the intermediate coordinate data of each target object after data format conversion to the panoramic video surveillance module also involves a real-time update process, that is, it is necessary to obtain new position data from the radar module regularly or in real time, and synchronize this data to the panoramic video surveillance module to ensure that the video image obtained by the panoramic video surveillance module matches the radar data.

[0191] The following is a detailed introduction to the process of real-time updating of the target object's position data.

[0192] Specifically, a separate thread or process can be set up in the radar module to continuously acquire real-time radar data from the radar. The acquired data is first determined to be valid, and invalid data is discarded. The radar module processes the radar data and obtains the position data of each target object. The radar module can then timestamp the current position data of each target object and send this timestamped position data to the central control module in real time.

[0193] The central control module receives the time-stamped position data of each target object, performs data conversion on the position data of each target object, and then stores the converted, time-stamped intermediate coordinate data of each target object in a buffer. It should be noted that the buffer can be an area in memory or a special data structure such as a ring buffer or a queue.

[0194] When the panoramic video surveillance module needs to obtain the intermediate coordinate data of each target object, it can directly read the required data from this buffer. To avoid data conflicts or loss, a synchronization mechanism such as a mutex, semaphore, or conditional variable is generally used. The data in the buffer is timestamped and stored to ensure data continuity and integrity.

[0195] Alternatively, the middleware in the central control module can also broadcast the intermediate coordinate data of each target object to the panoramic video monitoring module in real time. In order to implement the broadcast function, the middleware generally uses a publish-subscribe model or a message queue.

[0196] At the same time, in the panoramic video monitoring module, one or more threads continuously read the intermediate coordinate data of each target object from the buffer. After data processing, the intermediate coordinate data of each target object is finally displayed in the video image screen.

[0197] When data volumes are large or read and write operations are frequent, the central control module and panoramic video surveillance module use databases or message queues for data caching and storage. For example, time series databases are particularly well-suited for storing and processing timestamp-laden data, facilitating data queries and statistics.

[0198] During the entire real-time synchronization process, since both real-time and concurrency are required, this embodiment can be implemented using technologies such as multi-threading and asynchronous I / O.

[0199] Through the above method, after the intermediate coordinate data of each target object is transmitted to the panoramic video monitoring module in real time, the panoramic video monitoring module will perform certain data processing on the intermediate coordinate data of each target object, and then mark each target object in the panoramic video image based on the processed data.

[0200] It's worth noting that TCP / IP can be used for data transmission between the radar module, central control module, and panoramic video surveillance module. TCP / IP ensures data reliability and integrity. In practical applications, using TCP ensures real-time data delivery. Furthermore, TCP / IP includes a confirmation-reply mechanism. After sending data, the sender waits for a confirmation reply from the receiver. If no reply is received within a specified time, the sender resends the data.

[0201] The following is a detailed introduction to the process of labeling the panoramic video surveillance module.

[0202] Figure 7 This is the fourth flowchart of the airport panoramic monitoring method according to an embodiment of the present application.

[0203] like Figure 7 As shown, in some embodiments, the panoramic video monitoring module marks each target object in the video image according to the converted position data of each target object, which may include the following steps:

[0204] Step 710: Determine the geographic coordinate data of each target object according to the intermediate coordinate data of each target object.

[0205] Step 720: performing coordinate conversion on the geographic coordinate data of each target object to obtain the plane coordinate data of each target object. The plane coordinate data of the target object is the position coordinate of the target object in the coordinate system of the video image.

[0206] Step 730: Mark each target object in the video image according to the plane coordinate data of each target object.

[0207] It should be noted that the geographic coordinate data of the target object is the geographic coordinates of the target object relative to the earth, and the geographic coordinate data may include latitude, longitude and altitude information of the target object.

[0208] In step 710, the geographic coordinate data of each target object can be determined based on the intermediate coordinate data of each target object as follows: the earth is a sphere, and with the earth as a reference point, the intermediate coordinate data of the target object can be converted into geographic coordinate data through the following plane and spherical geometry conversion formula.

[0209] The conversion formula for longitude can be:

[0210] Δλ = X' / (R * cosφ1)

[0211] The conversion formula for latitude can be:

[0212] Δφ = Y' / R

[0213] Where Δλ is the longitude difference between the reference point and the target object, Δφ is the latitude difference between the reference point and the target object, φ1 is the latitude of the reference point, R is the radius of the earth, and (X', Y') is the intermediate coordinate data of the target object.

[0214] After obtaining the longitude and latitude differences between the reference point and the target object, the calculated longitude and latitude differences are added to the longitude and latitude of the reference point to obtain the geographic coordinate data of the target object.

[0215] Furthermore, because the geographic coordinate data of each target object needs to be matched with the video image coordinates obtained by the panoramic video surveillance module in order to mark the target object on the panoramic video image, it is also necessary to convert the geographic coordinate data of each target object into plane coordinate data under the video image coordinate system.

[0216] In some embodiments, the conversion of the target object's geographic coordinate data into plane coordinate data can be achieved using the following formula:

[0217]

[0218] in, is the plane coordinate data of the target object, R is the radius of the earth, λ is the longitude in the geographic coordinate data of the target object, and φ is the latitude in the geographic coordinate data of the target object.

[0219] It is understandable that the geographic coordinate data and plane coordinate data of the target object have different physical meanings. The geographic coordinate data describes the position of the target object on the surface of the earth, while the plane coordinate data describes the position of the target object on the image array. This conversion step is to accurately mark the actual position of the target object scanned by the radar on a certain frame of panoramic video image.

[0220] After obtaining the plane coordinate data of each target object, the panoramic video monitoring module can mark each target object on the video image based on the plane coordinate data of each target object. In this embodiment, each target object can be marked using only graphic annotation, only text labels, or a combination of graphic and text labels.

[0221] The following describes the labeling process of the panoramic video surveillance module in detail by taking the combination of graphic labeling and text labeling to label each target object as an example.

[0222] In some embodiments, after the radar module processes radar data to obtain position data for each target object in the airport, the method for panoramic airport monitoring may further include: the radar module acquiring basic information for each target object and transmitting the basic information for each target object to the central control module; the central control module synchronously transmitting the basic information for each target object to the panoramic video monitoring module. Labeling each target object in the video image based on the plane coordinate data of each target object may include: graphically labeling each target object in the video image using the circle function based on the plane coordinate data of each target object, and textually labeling each target object in the video image using the putText function based on the basic information of each target object.

[0223] Specifically, after detecting each target object, if the target object is a large aircraft, the radar module can establish a data link with the ATC (Air Traffic Control) system based on the flight plan to obtain basic information such as the target object's flight number and estimated landing time. Before the aircraft takes off, the flight plan is submitted to the relevant agencies and includes information such as the aircraft's flight number, takeoff time, and estimated flight path. The radar module can compare the real-time detection data with the flight plan to determine which flight plan matches the target object currently detected by the radar. It can then retrieve the matching flight plan to obtain the basic information of the target object in the flight plan.

[0224] Additionally, aircraft are typically equipped with a transponder, which automatically responds to radar signals by transmitting a signal containing the aircraft's identity, real-time flight parameter codes, and other data. Upon receiving this signal, the radar module can directly obtain basic information about the aircraft. This information exchange using a transponder is known as a secondary radar or SSR (Secondary Surveillance Radar) system.

[0225] After the radar module obtains the basic information of each target object through any of the above methods, it is also synchronously transmitted to the panoramic video monitoring module through the central control module. After the panoramic video monitoring module receives the basic information of each target object sent by the central control module, it can draw a point in the video image based on the plane coordinate data of each target object and use the circle function in the OpenCV algorithm. This point is used to represent the target object. Furthermore, the panoramic video monitoring module can add a text label at this point based on the basic information of each target object and use the putText function in the OpenCV algorithm. For example, the program code for annotation using the OpenCV algorithm can be:

[0226] cv2.circle(img, ,radius,color,thickness)

[0227] cv2.putText(img,text, ,font,size,color,thickness)

[0228] Among them, img is the video image, is the plane coordinate data of the target object, radius is the radius of the circle to be drawn, color is the color, thickness is the line width, text is the text to be added, font is the font, and size is the font size. To annotate multiple target objects, you can use a loop.

[0229] It should be noted that if the target object is an aircraft, the text label can include information such as the aircraft's altitude, flight number, aircraft type (such as A380, B747, etc.), aircraft status (such as flight status, taxiing status, etc.), aircraft speed (such as current speed or expected speed), and even the aircraft's destination or origin. The specific information to be labeled depends on the information needs of tower staff and flight management personnel.

[0230] In this embodiment, the panoramic video monitoring module also needs to update the video image and the position of the target object in the video image in real time. Specifically, the main program of the panoramic video monitoring module can continuously read the latest intermediate coordinate data of each target object from the buffer and update the annotation information and position of the target object in the latest video image.

[0231] In this embodiment, the panoramic video surveillance module may also include a user terminal with a display that can display the annotated video images. It should be noted that user terminals may include (but are not limited to) mobile terminals such as mobile phones, laptops, and PADs (tablet computers), as well as fixed terminals such as digital TVs and desktop computers. User terminals are not specifically defined here. The display process primarily includes two sub-steps: image rendering and image playback.

[0232] Image rendering converts video image data, which contains annotations of target object locations, into an image that can be displayed on a monitor. This process utilizes rendering techniques from computer graphics, such as lighting models and texture mapping. Image rendering can be accelerated by using a graphics processing unit (GPU) or graphics libraries such as OpenGL, Vulkan, or DirectX.

[0233] Image playback involves displaying rendered images continuously on a monitor to form a continuous video. This process typically requires video playback technologies, such as video decoding and frame rate control. Existing video playback libraries, such as FFmpeg and VLC, are used to implement video playback.

[0234] Furthermore, the display can also be used to display a user interface, allowing users to control video playback through the interface, such as starting / pausing playback, fast forward / rewinding, and changing playback speed. Furthermore, the user interface can also provide various auxiliary functions, such as video search, screenshots, and recording. The user interface can be implemented using a graphical user interface (GUI) library, such as Qt, GTK, or WxWidgets.

[0235] Thus, the radar module detects each target object in the airport, and by performing rain and snow denoising on the radar data, the interference of rain and snow on the radar data is eliminated, thereby obtaining more accurate target object location data; the central control module transmits the location data of each target object to the panoramic video monitoring module in real time and synchronously, ensuring the synchronization of radar data and panoramic video image data; the panoramic video monitoring module visually annotates each target object detected by the radar in the video image, and the annotated video image is displayed through the user terminal. Even in rainy, foggy and other weather conditions, the airport tower monitoring personnel can clearly and completely see the relevant information of aircraft or other large objects in the airport through the user terminal, thereby effectively improving the effectiveness and safety of airport operations. For example, through the airport panoramic monitoring system of this embodiment, the tower staff can see the exact location and taxiing speed of the aircraft preparing to depart, thereby making more accurate arrangements and commands.

[0236] Based on the above embodiments, the present invention further provides a computer program product, comprising a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the method performed by the radar module in the airport panoramic monitoring method provided by each of the above methods, including: obtaining radar data of the current airport; processing the radar data to obtain position data of each target object in the airport; and transmitting the position data of each target object to a central control module. Alternatively, the computer can execute the method performed by the central control module in the airport panoramic monitoring method provided by each of the above methods, including: performing data conversion on the position data of each target object and synchronously outputting the converted position data of each target object to a panoramic video monitoring module. Alternatively, the computer can execute the method performed by the panoramic video monitoring module in the airport panoramic monitoring method provided by each of the above methods, including: obtaining a video image of the current airport; and marking each target object in the video image based on the converted position data of each target object.

[0237] On the basis of the above embodiments, on another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the method executed by the radar module in the airport panoramic monitoring method provided by the above methods, the method comprising: obtaining radar data of the current airport; processing the radar data to obtain the position data of each target object in the airport, and sending the position data of each target object to the central control module. Alternatively, the method executed by the central control module in the airport panoramic monitoring method provided by the above methods can be executed, the method comprising: performing data conversion on the position data of each target object, and synchronously outputting the converted position data of each target object to the panoramic video monitoring module. Alternatively, the method executed by the panoramic video monitoring module in the airport panoramic monitoring method provided by the above methods can be executed, the method comprising: obtaining a video image of the current airport; marking each target object in the video image according to the converted position data of each target object.

[0238] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0239] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

Claims

1. An airport panoramic monitoring system, characterized in that: It includes a radar module, a panoramic video monitoring module and a central control module, wherein the central control module is respectively connected to the radar module and the panoramic video monitoring module in communication; The radar module is used to obtain radar data of the current airport, perform data processing on the radar data, obtain position data of each target object in the airport, and send the position data of each target object to the central control module; The central control module is used to convert the position data of each target object and synchronously output the converted position data of each target object to the panoramic video monitoring module; The panoramic video monitoring module is used to obtain the current video image of the airport and mark each target object in the video image according to the converted position data of each target object; The radar module is specifically used for: Converting the radar data into a radar image, and obtaining a grayscale value change rate of each pixel in the radar image; When the gray value change rate of the pixel point is less than a preset threshold, the pixel point is determined as a noise point; The noise points are denoised according to the sign of the grayscale value change rate of the noise points; when the grayscale value change rate of the noise points is positive, the grayscale value of the noise points is retained; when the grayscale value change rate of the noise points is negative, the grayscale value of the noise points is removed.

2. The airport panoramic monitoring system according to claim 1, characterized in that: The position data of the target object includes polar coordinate data of the target object, and the central control module includes middleware, and the central control module is specifically used for: Performing coordinate conversion on the polar coordinate data of each target object to obtain intermediate coordinate data of each target object in the coordinate system of the panoramic video monitoring module; Using the middleware, converting the data format of the intermediate coordinate data of each target object into a target data format, wherein the target data format is a data format corresponding to the panoramic video monitoring module; The intermediate coordinate data of each target object after data format conversion is synchronously transmitted to the panoramic video monitoring module.

3. The airport panoramic monitoring system according to claim 2, characterized in that: The panoramic video monitoring module is specifically used for: Determining geographic coordinate data of each target object according to the intermediate coordinate data of each target object; Performing coordinate conversion on the geographic coordinate data of each target object to obtain plane coordinate data of each target object, wherein the plane coordinate data of the target object is the position coordinate of the target object in the coordinate system of the video image; Each target object is marked in the video image according to the plane coordinate data of each target object.

4. The airport panoramic monitoring system according to claim 3, characterized in that: The radar module is further specifically used to obtain basic information of each target object and send the basic information of each target object to the central control module; The central control module is also specifically used to synchronously send the basic information of each target object to the panoramic video monitoring module; The panoramic video monitoring module is also specifically used to use the circle function and based on the plane coordinate data of each target object, to graphically annotate each target object in the video image, and to use the putText function and based on the basic information of each target object, to textually annotate each target object in the video image.

5. The airport panoramic monitoring system according to claim 1, characterized in that: The radar module includes at least one radar, and the panoramic video monitoring module includes at least one camera device; Each of the radars is used to obtain the radar data of the current airport; Each of the camera devices is used to obtain the video image of the current airport; The central control module is also used to perform time start point alignment and data frame synchronization processing on each of the radars and each of the camera devices.

6. The airport panoramic monitoring system according to claim 4, characterized in that: The panoramic video monitoring module also includes a display, and the display is used to display the video image after the graphic annotation and the text annotation.

7. A method for panoramic monitoring of an airport, characterized in that: Applied to the airport panoramic monitoring system according to any one of claims 1 to 6, the method comprises: The panoramic video monitoring module obtains the video image of the current airport; The radar module obtains radar data of the current airport, processes the radar data, obtains position data of each target object in the airport, and sends the position data of each target object to the central control module; The central control module performs data conversion on the position data of each target object, and synchronously outputs the converted position data of each target object to the panoramic video monitoring module; The panoramic video monitoring module marks each of the target objects in the video image according to the converted position data of each of the target objects; The step of processing the radar data to obtain the position data of each target object in the airport includes: Converting the radar data into a radar image, and obtaining a grayscale value change rate of each pixel in the radar image; When the gray value change rate of the pixel point is less than a preset threshold, the pixel point is determined as a noise point; The noise points are denoised according to the sign of the grayscale value change rate of the noise points; when the grayscale value change rate of the noise points is positive, the grayscale value of the noise points is retained; when the grayscale value change rate of the noise points is negative, the grayscale value of the noise points is removed.

8. The airport panoramic monitoring method according to claim 7, characterized in that: The position data of the target object includes polar coordinate data of the target object; The step of converting the position data of each target object comprises: Performing coordinate conversion on the polar coordinate data of each target object to obtain intermediate coordinate data of each target object in the coordinate system of the panoramic video monitoring module; Converting the data format of the intermediate coordinate data of each target object into a target data format, wherein the target data format is a data format corresponding to the panoramic video monitoring module; The intermediate coordinate data of each target object after data format conversion is synchronously transmitted to the panoramic video monitoring module.

9. The airport panoramic monitoring method according to claim 8, characterized in that: The step of marking each of the target objects in the video image according to the converted position data of each of the target objects includes: Determining geographic coordinate data of each target object according to the intermediate coordinate data of each target object; Performing coordinate conversion on the geographic coordinate data of each target object to obtain plane coordinate data of each target object, wherein the plane coordinate data of the target object is the position coordinate of the target object in the coordinate system of the video image; Each target object is marked in the video image according to the plane coordinate data of each target object.

10. The airport panoramic monitoring method according to claim 9, characterized in that: The panoramic video monitoring module converts the geographic coordinate data of the target object into the plane coordinate data of the target object by the following formula: in, is the plane coordinate data of the target object, R is the radius of the earth, λ is the longitude in the geographic coordinate data of the target object, and φ is the latitude in the geographic coordinate data of the target object.

11. The airport panoramic monitoring method according to claim 9, characterized in that: After the radar module processes the radar data to obtain the position data of each target object in the airport, the method further includes: The radar module acquires basic information of each target object and sends the basic information of each target object to the central control module; The central control module synchronously sends the basic information of each target object to the panoramic video monitoring module; The step of marking each target object in the video image according to the plane coordinate data of each target object includes: Using the circle function and according to the plane coordinate data of each target object, each target object is graphically annotated in the video image, and using the putText function and according to the basic information of each target object, each target object is textually annotated in the video image.

Images