Airport activity target sign display method and display system

Abstract

The present invention discloses a method and system for displaying signs of active targets at an airport. The method is applied to the airport scene, and includes: obtaining a panoramic video, identifying the active targets at the scene based on the panoramic video, and locating the three-dimensional coordinates of the active targets at the scene; collecting multi-source positioning data of the active targets at the scene, parsing the sign information from the multi-source positioning data; fusing the three-dimensional coordinates of the active targets at the scene with the sign information to form fused information, and projecting the fused information onto the panoramic image. The technical solution of the present application can more effectively conduct comprehensive situational awareness of the airport scene.

Description

Technical Field

[0001] The invention relates to the technical field of airport monitoring, and in particular to an airport movable target sign display method and an airport movable target sign display system. Background Art

[0002] In the tower control application of the airport, accurate situation information of the airport scene can help improve the situational awareness of the controller. Overlaying signs on all targets on the panoramic video of the airport scene is a basic and effective means. It can provide airport and air traffic control departments with airport scene implementation situation information, improve control efficiency, and effectively reduce the probability of safety accidents at the airport scene.

[0003] However, the current methods for monitoring situation information at airport scenes are relatively simple, making it difficult to conduct comprehensive situation awareness at airport scenes. Summary of the invention

[0004] In view of the defects in the prior art, the present invention provides a method for displaying airport activity target signs, which can more effectively provide comprehensive situation awareness of the airport scene.

[0005] The present application provides a method for displaying a signboard of an active target at an airport, which is applied to an airport scene and comprises:

[0006] Acquire a panoramic video, identify a scene moving target based on the panoramic video, and locate the three-dimensional coordinates of the scene moving target;

[0007] Collecting multi-source positioning data of the scene activity target, and parsing the sign information from the multi-source positioning data;

[0008] The three-dimensional coordinates of the scene activity target and the sign information are fused to form fused information, and the fused information is projected on the panoramic image.

[0009] In one aspect, the step of identifying a scene moving target based on the panoramic video and locating the three-dimensional coordinates of the scene moving target comprises:

[0010] Acquire an original input image from the panoramic video;

[0011] Identifying the category of the scene activity target in the original input image, and determining the two-dimensional coordinates of the scene activity target on the image;

[0012] The two-dimensional coordinates of the scene activity target are converted into three-dimensional coordinates by using a preset first conversion parameter.

[0013] In one aspect, the step of identifying the category of the scene activity target in the original input image and determining the two-dimensional coordinates of the scene activity target on the original input image comprises:

[0014] The target in the original input image is identified based on the target detection model, the category of the scene activity target is identified, and the two-dimensional coordinates of the scene activity target on the original input image are determined.

[0015] In one aspect, before the step of identifying the target in the original input image based on the target detection model, the step includes:

[0016] Collect historical videos of airport scenes under different conditions;

[0017] Using time series sampling to filter out redundant images in the historical video to obtain training images;

[0018] Marking the category of the scene activity target and the two-dimensional coordinates of the scene activity target on the training image to form annotation data, and the training image and the corresponding annotation data form a training sample;

[0019] The training samples are collected to form a training set, and based on the initial training model, the initial training model is trained iteratively based on the training set to obtain a target detection model.

[0020] In one aspect, after the step of locating the three-dimensional coordinates of the scene activity target, the method further comprises:

[0021] Acquire a surveillance target, and calculate a measured distance between the surveillance target and a scene activity target on the panoramic video;

[0022] If the measured distance is less than a preset threshold, the monitored target is associated with the scene activity target;

[0023] If the measured distance is greater than or equal to a preset threshold, the surveillance target is not associated with the scene activity target.

[0024] In one aspect, the step of fusing the three-dimensional coordinates of the scene activity target with the sign information to form fused information, and projecting the fused information on the panoramic image comprises:

[0025] The panoramic video and the multi-source positioning data are updated regularly at the same frequency, and the multi-source positioning data and the three-dimensional coordinates of the scene activity targets are superimposed together.

[0026] In one aspect, after the step of superimposing the multi-source positioning data and the three-dimensional coordinates of the scene activity target, the method further comprises:

[0027] The three-dimensional coordinates of the scene activity target and the multi-source positioning data are fused in a weighted manner according to the lens distance for shooting the panoramic video.

[0028] In one aspect, the step of fusing the three-dimensional coordinates of the scene activity target and the multi-source positioning data in a weighted manner according to the lens distance of shooting the panoramic video includes:

[0029] For the scene moving targets with a close lens distance, the positioning weight of the three-dimensional coordinates of the scene moving targets in the panoramic video is increased;

[0030] For scene moving targets at a long distance from the camera, the positioning weight of the multi-source positioning data is increased.

[0031] In one aspect, the step of projecting the fusion information onto the panoramic image further includes:

[0032] Based on a preset second conversion parameter, the coordinate points in the fusion information are projected onto the panoramic image.

[0033] In addition, in order to solve the above problems, the present application also provides an airport movable target sign display system, the airport movable target sign display system is applied to the airport scene, and the display system includes:

[0034] An acquisition module, wherein the acquisition module identifies a scene activity target based on the panoramic video and locates the three-dimensional coordinates of the scene activity target;

[0035] A parsing module, the parsing module is used to collect multi-source positioning data of the scene activity target and parse the sign information from the multi-source positioning data;

[0036] A fusion module is used to fuse the three-dimensional coordinates of the scene activity target with the sign information to form fusion information, and project the fusion information on the panoramic image.

[0037] The beneficial effects of the present invention are as follows: locating the scene activity target from the panoramic video and determining the three-dimensional coordinates of the scene activity target. Collecting multi-source positioning data of the scene activity target improves the situational awareness of the airport scene. By fusing the three-dimensional coordinates of the scene activity target with the sign information in the multi-source positioning data and projecting the fused information on the panoramic image, the sign information can be observed more intuitively through the scene activity target, thereby more effectively achieving comprehensive situational awareness of the airport scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following is a brief introduction to the drawings required for the specific embodiments or the description of the prior art. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn according to the actual scale.

[0039] Figure 1 A schematic diagram of the process steps of the airport activity target sign display method in this application;

[0040] Figure 2 A schematic diagram of the process steps for converting two-dimensional coordinates into three-dimensional coordinates in the airport activity target sign display method of this application;

[0041] Figure 3 A schematic diagram of the process steps for obtaining two-dimensional coordinates through a target detection model in the airport activity target sign display method of this application;

[0042] Figure 4 A schematic diagram of the process steps for obtaining a target detection model in the airport activity target sign display method of this application;

[0043] Figure 5 A schematic diagram of the process steps for associating a surveillance target with a scene activity target on an image in the airport activity target sign display method of the present application;

[0044] Figure 6 A schematic diagram of the process steps of superimposing multi-source positioning data on three-dimensional coordinates and projecting the fused information on a panoramic image in the airport activity target sign display method of this application;

[0045] Figure 7 A schematic diagram of the process steps for allocating positioning weights of different data sources in the airport activity target sign display method of this application;

[0046] Figure 8 This is a schematic diagram of the module structure of the airport activity target sign display system for this application;

[0047] Fig. 9 This is a schematic diagram of the signage of the scene activity target in the panoramic image in the airport activity target signage display method of this application.

[0048] In the accompanying drawings, 101 is a sign; 100 is an acquisition module; 200 is a parsing module; 300 is a fusion module; and 400 is an offline panoramic camera calibration module. DETAILED DESCRIPTION

[0049] The following embodiments of the technical solution of the present invention are described in detail in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and are therefore only used as examples, and cannot be used to limit the protection scope of the present invention.

[0050] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in this application should have the common meanings understood by those skilled in the art to which the present invention belongs.

[0051] like Figure 1 As shown, the present application provides a method for displaying an active target sign at an airport. The method for displaying an active target sign at an airport is applied to the airport scene, which mainly includes the area for taxiing on the ground and the airspace above the airport where low-altitude flight is possible. The method for displaying an active target sign at an airport includes:

[0052] Step S10, obtain panoramic video, identify the scene moving targets based on the panoramic video, and locate the three-dimensional coordinates of the scene moving targets; panoramic video is a video shot with a 3D camera in all directions at 360 degrees. After shooting, these images will be post-processed to form a video that can realize the three-dimensional space display function. Users can adjust the up, down, left, and right directions of the video at will when watching. The panoramic video can improve the coverage of the airport scene and ensure that moving targets in any direction can be covered. The moving target can be an aircraft, a vehicle, etc., or other moving targets.

[0053] Furthermore, the height of the aircraft can be obtained by using the three-dimensional coordinates obtained by positioning in step S10, so that the position of a low-altitude aircraft can also be determined.

[0054] Step S20, collect multi-source positioning data of scene activity targets, and parse sign information from the multi-source positioning data; multi-source positioning data refers to monitoring data other than panoramic video, and multi-source positioning data mainly includes: scene monitoring radar (field monitoring radar), broadcast automatic dependent surveillance system (ADS-B) and multi-point positioning system (MLAT), etc. By collecting multi-source positioning data, the single source of positioning data is avoided, and more accurate positioning of scene activity targets can be provided. Sign information is mainly used to display aircraft information. Sign information may include address code and flight number, etc. The content of sign information can be added or deleted as needed. It should be noted that multi-source positioning data includes different data categories.

[0055] Step S30, the three-dimensional coordinates of the scene activity target and the sign information are merged to form fusion information, and the fusion information is projected on the panoramic image. The three-dimensional coordinates of the scene activity target are mainly used to determine the location of the scene activity target. By projecting the fusion information on the panoramic image, the airport scene situation can be observed more intuitively.

[0056] In this embodiment, the scene moving target is located from the panoramic video, and the three-dimensional coordinates of the scene moving target are determined. Multi-source positioning data of the scene moving target is collected to improve the situational awareness of the airport scene. By fusing the three-dimensional coordinates of the scene moving target with the sign information in the multi-source positioning data, and projecting the fused information on the panoramic image, the sign information can be observed more intuitively through the scene moving target, thereby more effectively achieving comprehensive situational awareness of the airport scene.

[0057] Furthermore, by fusing multi-source positioning data and correlating targets on the airport surface, accurate and reliable target sign prompt information can be provided for apron and tower control, effectively enhancing the situational awareness of controllers, providing strong support for controllers to accurately perceive possible risks in airport traffic, and improving the safety prevention capabilities and work efficiency in control work.

[0058] like Figure 2 As shown, in one embodiment of the present application, the steps of identifying a scene moving target based on a panoramic video and locating the three-dimensional coordinates of the scene moving target include:

[0059] Step S110, obtaining an original input image from the panoramic video; the original input image is an image including the aircraft. Since the panoramic video includes multiple frames of images per second, the original input image can be obtained by extracting the panoramic video frame by frame.

[0060] Step S120, identifying the category of the scene activity target in the original input image, and determining the two-dimensional coordinates of the scene activity target on the original input image; there are many types of aircraft, such as passenger planes, helicopters and transport planes, and different types of aircraft have different dimensions. The original input image has pixel coordinates, and the two-dimensional coordinates on the original input image can be understood as pixel coordinates.

[0061] Step S130, converting the two-dimensional coordinates of the scene activity target into three-dimensional coordinates by using a preset first conversion parameter. The three-dimensional coordinates can be understood as the position in the world coordinate system. The first conversion parameter can be used to convert between the world coordinates and the pixel coordinates. Converting the two-dimensional coordinates into three-dimensional coordinates can more accurately reflect the spatial position of the scene activity target.

[0062] like Figure 3 As shown, in one embodiment of the present application, the steps of identifying the category of the scene activity target in the original input image and determining the two-dimensional coordinates of the scene activity target on the original input image include:

[0063] Step S121, based on the target detection model, the target in the original input image is identified, the category of the scene activity target is identified, and the two-dimensional coordinates of the scene activity target on the original input image are determined. The target detection model can be a large language model or a large world model. The target detection model can realize automatic judgment of the original input image, reduce the error of human judgment, and improve the detection efficiency. For example, the target detection model can be YOLOv8, and other target detection models can also be used, including but not limited to the yolo series, R-CNN series, DETR, SSD, etc.

[0064] like Figure 4 As shown, in one embodiment of the present application, before the step of identifying the target in the original input image based on the target detection model, the following steps are included:

[0065] Step S101, collect historical videos of airport scenes under different conditions; the different conditions here include different weather conditions and different lighting conditions. Weather and lighting have a direct impact on the picture of the panoramic video, affecting the brightness and even clarity of the picture. The panoramic video of the airport scene is usually stored in a storage device, and the video with mainly moving targets is collected from the storage device. A rough screening can be performed to actively select historical video data under different weather and lighting conditions.

[0066] Step S102, using time series sampling to filter out redundant images in historical videos to obtain training images; historical videos are long and have a high number of frames, which contain a large amount of image information. In order to improve efficiency, some redundant images can be filtered out. In this way, representative training images can be obtained more quickly. The process of filtering out redundant images can be completed automatically by instruments and equipment, or by human-machine cooperation to manually calibrate the filtering results.

[0067] Step S103, marking the category and two-dimensional coordinates of the scene activity target on the training image to form annotation data, and the training image and the corresponding annotation data form a training sample; the category and position of the scene activity target can be automatically annotated, for example, the collected training image is automatically annotated using a target detection model. In addition, the automatically annotated data may be incorrectly annotated due to missed detection or false detection. Therefore, after the category and position of the target are annotated, the annotation can be corrected. The correction method can be manual correction, machine correction, or manual and machine interaction to complete the correction together.

[0068] Step S104, collect training samples to form a training set, based on the initial training model, and perform training iterations on the initial training model based on the training set to obtain a target detection model. In this case, each labeled data corresponds to a training image, and the two form a pair of training samples, and the training set includes the training samples. By training the initial training model, fine-tuning of the initial training model is achieved, thereby obtaining a target detection model.

[0069] The specific training process for the target detection model can be: during training, the initial learning rate is set to 1E-3, and the learning rate warm-up method is used. In the first iteration, the learning rate is 0.1 times the initial learning rate, and in the second iteration, the learning rate is 0.2 times the initial learning rate, and so on, until the initial learning rate is reached. When the preset number of iterations is reached, the adaptive moment estimation method (ADAM) can also be used to optimize the initial training model. During inference, tensorRT is combined to improve inference efficiency and control the target detection time to within 40ms to achieve real-time target detection function.

[0070] like Figure 5 As shown, in one embodiment of the present application, after the step of locating the three-dimensional coordinates of the scene activity target, the following steps are included:

[0071] Step S131, obtain the surveillance target, calculate the measured distance between the surveillance target and the scene activity target on the panoramic video; it can also be understood as the distance between the surveillance target position and the three-dimensional coordinates of the scene activity target. Usually, there are many aircrafts on the airport scene, and it is necessary to correspond the calculated three-dimensional coordinates with the surveillance target in the panoramic video. In this embodiment, the correspondence between the three-dimensional coordinates and the surveillance target is mainly determined by the size of the distance.

[0072] Step S131a, if the measured distance is less than the preset threshold, the surveillance target is associated with the surface activity target; different aircrafts have different sizes, and the size of the preset threshold can be adjusted according to the type of the surface activity target to make it more suitable for the size of the aircraft. For example, for a passenger aircraft, the preset threshold can be 20 meters. If the measured distance is less than the preset threshold, it means that the three-dimensional coordinates of the surveillance target and the surface activity target refer to the same object.

[0073] Step S131b, if the measured distance is greater than or equal to the preset threshold, the monitoring target and the scene activity target are not associated. If the measured distance is greater than or equal to the preset threshold, it means that the three-dimensional coordinates of the monitoring target and the scene activity target refer to two different objects, and the two are unrelated and are not associated.

[0074] like Figure 6 As shown, in one embodiment of the present application, the steps of fusing the three-dimensional coordinates of the scene activity target with the sign information to form fusion information, and projecting the fusion information on the panoramic image include:

[0075] Step S310, the panoramic video and multi-source positioning data are updated regularly at the same frequency, and the multi-source positioning data and the three-dimensional coordinates of the scene activity targets are superimposed together. Due to the inconsistent sending and receiving frequencies of different data sources, the time for updating the multi-source positioning data is also not uniform. For example, the panoramic video is generally updated 25 times per second, and the comprehensive track is updated once per second. In order to keep the frequency uniform when different data sources are merged, the panoramic video and multi-source positioning data are sent regularly 25 times per second. Furthermore, this embodiment can use Kalman filtering to reduce the interference of noise signals. Multi-source positioning data generally conforms to specific format specifications, such as receiving multicast data through the UDP protocol, and using asterix4py to parse the corresponding CAT62 format message to obtain the target geographic coordinate system position, flight number, address code and other information.

[0076] In one embodiment of the present application, after the step of superimposing the multi-source positioning data and the three-dimensional coordinates of the scene activity target, the following steps are included:

[0077] Step S320, the three-dimensional coordinates of the scene moving target and the multi-source positioning data are fused in a weighted manner according to the lens distance of the panoramic video. A panoramic camera can be used to shoot the panoramic video, and different scene moving targets are at different distances from the camera lens. Different data sources contribute differently to the positioning of the target. The weight of the data source with accurate measurement can be increased, and the weight of the data source with low measurement accuracy can be reduced.

[0078] like Figure 7 As shown, in one embodiment of the present application, the step of fusing the three-dimensional coordinates of the scene activity target and the multi-source positioning data in a weighted manner according to the lens distance of the panoramic video shooting includes:

[0079] Step S321, for scene activity targets at a close lens distance, the positioning weight of the three-dimensional coordinates of the scene activity targets in the panoramic video is increased; the scene activity targets at a close lens distance are displayed more clearly in the panoramic video, and the three-dimensional coordinates of the scene activity targets calculated from the panoramic video are more accurate and have higher precision; the positioning weight of the three-dimensional coordinates of the scene activity targets is increased, and the weight of other multi-source positioning data is reduced, thereby making it possible to position the scene activity targets more accurately.

[0080] Step S322, for the scene moving target with a long lens distance, increase the positioning weight of the multi-source positioning data. The scene moving target with a long lens distance is not displayed with high clarity in the panoramic video, the uncertainty of the scene moving target increases, and the accuracy of the three-dimensional coordinates of the scene moving target calculated thereby decreases accordingly. At this time, the positioning weight of the three-dimensional coordinates of the scene moving target can be reduced, and the weight of other multi-source positioning data can be increased, so that the positioning of the distant target can be made more accurate.

[0081] The Fermi function can be used to estimate the weight coefficients for weight calculation.

[0082]

[0083] Where d represents the distance from the monitored target to the camera, μ represents the distance threshold, and β is an adjustable coefficient. For the same target, the weight coefficients obtained from different data sources will eventually be normalized to the range of [0, 1]. For the two data sources of panoramic video and multi-source positioning data, assuming that X represents the fused position, w represents the normalized weight, the subscript υ represents the panoramic video, and s represents the multi-source positioning data, the final weighted result is

[0084] X=w υ X υ +w s X s , where X can be the coordinate in any coordinate system including camera coordinate system, world coordinate system and geographic coordinate system.

[0085] like Figure 6 As shown, in one embodiment of the present application, the step of projecting the fusion information onto the panoramic image further includes:

[0086] Step S330: projecting the coordinate points in the fusion information onto the panoramic image based on a preset second conversion parameter. The second conversion parameter is mainly used to convert the three-dimensional coordinates into two-dimensional coordinates on the panoramic image.

[0087] The first conversion parameter and the second conversion parameter are both generated in advance. The following example illustrates the generation process of the first conversion parameter and the second conversion parameter:

[0088] (1) First, combine the high-definition map to collect ground landmarks that are clearly visible on the map and the panoramic image, and obtain the pixel coordinates and longitude and latitude coordinates of the calibration point pairs. Use RTK to collect spatial data as a supplement. Use RTK with a pole height of h to measure the longitude and latitude data. The coordinates of the two points at the bottom and top of the pole can be measured, and the pixel coordinates of the corresponding points can be found on the image. Measure k groups of points to obtain all spatial calibration points.

[0089] (2) Coordinate system definition: 2D pixel coordinate system, with its origin at the upper left corner of the image, the x-axis parallel to the horizontal direction of the image, and the y-axis parallel to the vertical direction of the image; 3D camera coordinate system, with its origin at the optical center of the camera, the x-axis parallel to the runway direction, the y-axis parallel to the gravity direction, and the z-axis as the camera optical axis, which is perpendicular to the imaging plane; 3D world coordinate system, defined as the geocentric earth-fixed coordinate system, with its origin O as the center of mass of the earth, the z-axis parallel to the earth's axis pointing to the North Pole, the x-axis pointing to the intersection of the prime meridian and the equator, and the y-axis perpendicular to the xOz plane to form a right-handed coordinate system; geographic coordinate system, a coordinate system that uses a three-dimensional spherical surface to define the position on the earth's surface, describing the points on the earth's surface by longitude and latitude.

[0090] (3) Coordinate system conversion: Assume that (x, y) represents pixel coordinates, (lon, lat, alt) represents coordinates in the corresponding geographic coordinate system, and the three elements represent longitude, latitude, and altitude respectively. The relationship between the two can be expressed by Formula 1:

[0091]

[0092] Where D represents the conversion from longitude and latitude to the world coordinate system, R and T represent the rotation and translation matrices from the world coordinate system to the camera coordinate system, S represents the splicing coefficient, and P represents the projection coefficient. Therefore, camera calibration is to determine the values ​​of the above parameters. R and T can be understood as the first preset parameters, and S and P can be understood as the second preset parameters.

[0093] (4) Estimate D: First, convert the longitude and latitude coordinates (lon, lat) into plane coordinates (x, y)

[0094]

[0095] Where r represents the radius of the earth. Based on the plane coordinates, the height information is added to obtain the coordinates of the target in the world coordinate system.

[0096] (5) Estimate R and T: First, based on the definition of the camera coordinate system, estimate the rotation matrix R and translation matrix T of the camera coordinate system relative to the world coordinate system. Use RTK to measure the positions of the calibration points and unify these points to the world coordinate system. First, assume that the position of the optical center of the panoramic camera is O c , then the translation matrix T = O c Next, select several discrete points at the center of the runway, obtain the spatial coordinates of these points, and use the least squares method to fit the vector R of the X-axis of the camera coordinate system x Randomly select some points on the apron and runway area, measure the coordinates of these points on the ground, and fit these points into a plane, with the upward direction of the normal of the plane as the Y-axis vector R y Using the right-hand rule, we can get the Z-axis vector R z.at last

[0097]

[0098] (6) Estimate P: P represents the projection of the camera coordinate system to the image coordinate system. Assume that the panoramic image is composed of n camera images, and use the projection parameter K to establish the association between the camera coordinates and the corresponding image coordinates. For the i-th (1≤i≤n) camera, assume (X C , Y C , Z C ) and (m x , m y ) represents a pair of camera coordinate points and pixel coordinate points, and the two have the following relationship:

[0099]

[0100] Where P i represents the projection matrix of the i-th virtual camera, and P i The camera focal length in the x and y directions, f x f y Coordinates of the principal point (c x , c y )constitute.

[0101] (7) Estimate S: The n cameras in (6) are estimated by the camera stitching coefficient S i (1≤i≤n) to obtain a panoramic image, so S i This can describe the mapping of a single camera image pixel coordinate to a panoramic image coordinate. First, the single image coordinate (x, y) is projected onto the cylindrical coordinate (x cyc ,y cyc ):

[0102]

[0103] Where f represents the focal length of the original image, w and h represent the width and height of the original image respectively. The two adjacent cylindrical projection images are stitched together. Assume that the two adjacent cylindrical projection images are I a and I b First, SIFT (Scale Invariant Feature Transform) is used to transform I a and I b Extract image features, obtain feature descriptors, and then use FLANN (Fast Approximate Nearest Neighbor Search) to find the corresponding feature points between the two images, and further use RANSAC (Random Sampling Consensus Algorithm) to further eliminate false matches and improve matching accuracy. Use affine transformation to establish the transformation relationship between matching points, and use the least squares method to solve the affine transformation matrix A. Assume that the pixel coordinates of the panoramic image are (x pano ,y pano),So

[0104]

[0105] Among them, (t x , t y ) represents a translation transformation, Represents a rotation transformation.

[0106] like Figure 8 As shown, the present application also provides an airport activity target sign display system, which is applied to the airport scene. The display system includes: an acquisition module 100, a parsing module 200 and a fusion module 300.

[0107] The acquisition module 100 identifies the target of the scene activity based on the panoramic video and locates the three-dimensional coordinates of the target of the scene activity; the panoramic video is a video shot in 360 degrees with a 3D camera. After the shooting is completed, these images will be post-processed to form a video that can realize the three-dimensional space display function. The user can adjust the up, down, left, and right directions of the video at will when watching. The panoramic video can improve the coverage of the airport scene and ensure that aircraft in any direction can be covered.

[0108] The parsing module 200 is used to collect a variety of multi-source positioning data for scene activity targets, and parse the sign information from the multi-source positioning data; the multi-source positioning data mainly includes: scene surveillance radar (field monitoring radar), broadcast automatic dependent surveillance system (ADS-B) and multi-point positioning system (MLAT), etc. By collecting multi-source positioning data, the single source of positioning data is avoided, and more accurate positioning of scene activity targets can be provided. The sign information is mainly used to display the information of the aircraft. The sign information may include the address code and flight number, etc. The content of the sign information may be added or deleted as needed. It should be noted that the multi-source positioning data includes different data categories.

[0109] The fusion module 300 is used to fuse the three-dimensional coordinates of the scene activity target with the sign information to form fusion information, and project the fusion information on the panoramic image. Projecting the fusion information on the panoramic image can more intuitively observe the airport scene. Fig. 9 As shown, it can be seen that a sign 101 is marked on the distant scene activity target.

[0110] The display system further includes an offline panoramic camera calibration module 400, which is used to pre-generate a first conversion parameter and a second conversion parameter, and save the first conversion parameter and the second conversion parameter.

[0111] The specific embodiments and beneficial effects of the airport activity target sign display system in the present application can be found in the above-mentioned airport activity target sign display method, which will not be repeated here.

[0112] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and specification of the present invention.

Claims

1. A method for displaying a signboard of an airport activity target, characterized in that: The airport moving target sign display method is applied to the airport scene, and the airport moving target sign display method comprises: Acquire a panoramic video, identify a scene moving target based on the panoramic video, and locate the three-dimensional coordinates of the scene moving target; Collecting multi-source positioning data of the scene activity target, and parsing the sign information from the multi-source positioning data; The three-dimensional coordinates of the scene activity target and the sign information are fused to form fused information, and the fused information is projected on the panoramic image.

2. The method for displaying airport activity target signs according to claim 1, characterized in that: The step of identifying a scene activity target based on the panoramic video and locating the three-dimensional coordinates of the scene activity target comprises: Acquire an original input image from the panoramic video; Identifying the category of the scene activity target in the original input image, and determining the two-dimensional coordinates of the scene activity target on the original input image; The two-dimensional coordinates of the scene activity target are converted into three-dimensional coordinates by using a preset first conversion parameter.

3. The method for displaying airport activity target signs according to claim 2, characterized in that: The step of identifying the category of the scene activity target in the original input image and determining the two-dimensional coordinates of the scene activity target on the original input image comprises: The target in the original input image is identified based on the target detection model, the category of the scene activity target is identified, and the two-dimensional coordinates of the scene activity target on the original input image are determined.

4. The method for displaying airport activity target signs according to claim 3, characterized in that: Before the step of identifying the target in the original input image based on the target detection model, the method includes: Collect historical videos of airport scenes under different conditions; Using time series sampling to filter out redundant images in the historical video to obtain training images; Marking the category of the scene activity target and the two-dimensional coordinates of the scene activity target on the training image to form annotation data, and the training image and the corresponding annotation data form a training sample; The training samples are collected to form a training set, and based on the initial training model, the initial training model is trained iteratively based on the training set to obtain a target detection model.

5. The method for displaying airport activity target signs according to claim 1, characterized in that: After the step of locating the three-dimensional coordinates of the scene activity target, the method further comprises: Acquire a surveillance target, and calculate a measured distance between the surveillance target and a scene activity target on the panoramic video; If the measured distance is less than a preset threshold, the monitored target is associated with the scene activity target; If the measured distance is greater than or equal to a preset threshold, the surveillance target is not associated with the scene activity target.

6. The method for displaying airport moving target signs according to claim 1, characterized in that: The step of fusing the three-dimensional coordinates of the scene activity target with the sign information to form fused information, and projecting the fused information onto the panoramic image comprises: The panoramic video and the multi-source positioning data are updated regularly at the same frequency, and the multi-source positioning data and the three-dimensional coordinates of the scene activity targets are superimposed together.

7. The method for displaying airport moving target signs according to claim 6, characterized in that: After the step of superimposing the multi-source positioning data and the three-dimensional coordinates of the scene activity target, the method further comprises: The three-dimensional coordinates of the scene activity target and the multi-source positioning data are fused in a weighted manner according to the lens distance for shooting the panoramic video.

8. The method for displaying airport moving target signs according to claim 7, characterized in that: The step of fusing the three-dimensional coordinates of the scene activity target and the multi-source positioning data in a weighted manner according to the lens distance for shooting the panoramic video comprises: For the scene moving targets with a close lens distance, the positioning weight of the three-dimensional coordinates of the scene moving targets in the panoramic video is increased; For scene moving targets at a long distance from the camera, the positioning weight of the multi-source positioning data is increased.

9. The method for displaying airport moving target signs according to any one of claims 1 to 8, characterized in that: The step of projecting the fusion information onto the panoramic image further includes: Based on a preset second conversion parameter, the coordinate points in the fusion information are projected onto the panoramic image.

10. An airport moving target sign display system, characterized in that: The airport active target sign display system is applied to the airport scene, and the display system comprises: An acquisition module, wherein the acquisition module identifies a scene activity target based on the panoramic video and locates the three-dimensional coordinates of the scene activity target; A parsing module, the parsing module is used to collect multi-source positioning data of the scene activity target and parse the sign information from the multi-source positioning data; A fusion module is used to fuse the three-dimensional coordinates of the scene activity target with the sign information to form fusion information, and project the fusion information on the panoramic image.

Images