Digital mapping method and device for airport runway aircraft running state, and storage medium

Abstract

The invention relates to a digital mapping method and device for an airport runway airplane running state and a storage medium. The method comprises the following steps that a plurality of images are acquired from a plurality of side lamp type image capturing devices on the two sides of an airport runway; for the image acquired by each image capture device, calculating the confidence of the same target category of the same grid in a plurality of adjacent frames, and weighting the confidence as the final confidence of the current frame to obtain the pixel coordinate information of the target in the image so as to realize target detection; based on the pixel coordinate information of the target in the image, utilizing a pre-constructed image coordinate-real coordinate conversion relation to obtain real coordinate information through conversion, and performing coordinate connection processing on the target located at the junction of the coverage area of the image capture equipment; and transmitting the real coordinate information subjected to coordinate connection processing into a pre-constructed digital runway to realize digital mapping of the target in the flight area. According to the invention, high-expansibility and low-cost state tracking for the target in the airport runway flight area is realized.

Description

Technical Field

[0001] The invention relates to the technical field of airport ground video monitoring, and in particular to a digital mapping method, equipment and storage medium for the running status of an aircraft on an airport runway. Background Art

[0002] By acquiring and analyzing the aircraft images in the airfield, the operating status of the aircraft in the airfield can be effectively obtained, including its real-time position, operating speed, offset and other information. It can provide the operating parameters of the aircraft and other targets for subsequent automated airfield monitoring, operational risk perception in the airfield, and real-time perception of digital runway construction, which is of great significance to airfield monitoring and digital runway construction.

[0003] The existing airport surface management system is mainly completed by the cooperation of primary surveillance radar, secondary surveillance radar, automatic dependent radar-broadcast (ADS-B), multi-lateration positioning (MLAT), surface surveillance radar, etc. These devices can provide aircraft operation information and provide guarantee for the safety of flight area operation, but they also have defects in accuracy and cost.

[0004] Chinese patent application publication number CN112102372A provides a technology for cross-camera trajectory tracking of airport ground objects, which utilizes the monitoring resources of all cameras in the airport, associates the same objects across scenes through multi-target tracking technology, realizes continuous trajectory tracking, and improves the stability and accuracy of target detection results. Although this method solves the problem of cross-camera target tracking to a certain extent.

[0005] However, the monitoring means of the above application are limited to providing monitoring functions, which are difficult to expand on their basis. The monitoring data is not combined with the digital runway to carry out related work on intelligent monitoring of the airfield. Summary of the invention

[0006] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a digital mapping method, device and storage medium for the operating status of aircraft on an airport runway. By transmitting monitoring data to the digital runway, highly scalable and low-cost status tracking of the airport runway flight area for targets can be achieved.

[0007] The purpose of the present invention can be achieved by the following technical solutions:

[0008] One aspect of the present invention provides a method for digitally mapping the operating status of an aircraft on an airport runway, comprising the following steps:

[0009] Step S1, acquiring multiple images from multiple edge light image capture devices on both sides of the airport runway;

[0010] Step S2, for each image captured by the image capture device, calculate the confidence of the same target category in the same grid in multiple adjacent frames and weight them as the final confidence of the current frame, obtain the pixel coordinate information of the target in the image, and realize target detection;

[0011] Step S3, based on the pixel coordinate information of the target in the image, using the pre-constructed image coordinate-real coordinate conversion relationship, converting to obtain real coordinate information, and performing coordinate connection processing on the target located at the intersection of the coverage range of the image capture device;

[0012] Step S4, the actual coordinate information after coordinate connection processing is transmitted to the pre-built digital runway to realize the digital mapping of the target in the flight area.

[0013] As a preferred technical solution, in step S2, the final confidence of the current frame is calculated using the following formula:

[0014]

[0015] Among them, conf final is the final confidence of the current frame, n is the memory range before the frame, and p i is the superposition weight of the previous i-th frame, conf i is the confidence of the previous i frames.

[0016] As a preferred technical solution, in step S3, the conversion relationship between image coordinates and real coordinates is constructed as follows:

[0017] X w =x / width*L eap +X c

[0018] Among them, X W is the horizontal coordinate of the world coordinate system, x is the horizontal coordinate of the image coordinate system, width is the pixel width of the image, L cap is the image capture device coverage, X c is the camera coordinate.

[0019] As a preferred technical solution, the coverage of the image capture device is calculated using the following formula:

[0020]

[0021] L cap =T cap *V t

[0022] Among them, T cap is the time the aircraft is within the coverage of the image capture device, N expis the number of frames expected by a single image capture device, FPS is the shooting frame rate, L cap Coverage for a single image capture device, V t is the takeoff speed.

[0023] As a preferred technical solution, in step S3, the process of performing coordinate connection processing on the target located at the boundary of the coverage area of ​​the image capture device includes:

[0024] Determine whether the distance between the center coordinates of the detection frames of two targets located at the intersection of the coverage range of the image capture device is less than a preset value. If so, merge the two targets into one target and update the coordinates of the target.

[0025] As a preferred technical solution, the merging of the two targets into one target and updating the coordinates of the target is implemented by the following formula:

[0026] (X3,Y3)=((X1+X2) / 2,(Y1+Y2) / 2)

[0027] Among them, (X1, Y1) and (X2, Y2) are the center coordinates of the detection box of the two targets before merging, and (X3, Y3) are the center coordinates of the detection box of the updated target.

[0028] As a preferred technical solution, the step S4 includes:

[0029] Step S401, matching the real coordinate information after the coordinate connection processing with the coordinates of the model in the scene at the previous moment, and judging whether the difference between the real coordinates and the coordinates of the model in the scene is less than a preset value, if so, updating the coordinates of the model in the scene to the real coordinates, if not, deleting the model in the scene;

[0030] Step S402, adding a model to the scene for the coordinates that are not matched in the real coordinate information;

[0031] Step S403: output the real-time operation information of the runway, including the position and size of the aircraft target.

[0032] Another aspect of the present invention provides a digital mapping device for the operation status of an aircraft on an airport runway, which is used to implement the digital mapping method for the operation status of an aircraft on an airport runway as described above, and the digital mapping device comprises:

[0033] A plurality of image capture devices are located on the same side of the runway and are arranged at equal intervals along the runway direction and are installed in an edge light style, with the viewing angle of the image capture devices parallel to the ground;

[0034] The data processing terminal is used to obtain images from the image capture device, and realize digital mapping of targets in the flight area through target detection, coordinate conversion and connection processing.

[0035] Another aspect of the present invention provides an electronic device, comprising: one or more processors and a memory, wherein the memory stores one or more programs, and the one or more programs include instructions for executing the aforementioned digital mapping method of the airport runway aircraft operating status.

[0036] Another aspect of the present invention provides a computer-readable storage medium comprising one or more programs for execution by one or more processors of an electronic device, wherein the one or more programs include instructions for executing the aforementioned method for digital mapping of the operating status of an aircraft on an airport runway.

[0037] Compared with the prior art, the present invention has at least one of the following beneficial effects:

[0038] (1) Realizing airport runway target status tracking based on multiple image capture devices: The present invention first acquires images from multiple image devices, then calculates the confidence of the current frame by memorizing the confidence of the previous and next frames to realize target recognition, then performs coordinate conversion and coordinate connection processing, and finally transmits the real coordinates to the pre-constructed digital runway to realize digital mapping of the targets in the flight area.

[0039] (2) High positioning accuracy: The present invention calculates the confidence of the current frame by memorizing the confidence of the previous and next frames, and transfers the confidence weighting of the same category of the same grid in the adjacent frame to the current frame, thereby enhancing the detection of the current frame by the adjacent frame information, ensuring that the positioning accuracy of the operation status acquisition method is high and meets the positioning accuracy requirements required for flight zone monitoring.

[0040] (3) Low cost of status tracking: The hardware of the present invention only needs to be equipped with multiple image capture devices, which is lower in cost than traditional monitoring methods, reduces construction costs, and can be promoted for use in small and medium-sized airports.

[0041] (4) Strong scalability: The present invention connects the mapping data to the digital runway, which is highly scalable. It can not only accurately display the real-time operation information of the runway, but also provide the real-time operation data required for flight zone risk perception and intelligent warning.

[0042] (5) The reality correspondence is intuitive: the present invention uses a camera as a sensing device to capture real-time images, and can conveniently check at any time whether the correspondence between the actual operating status and the mapping status is correct, which is beneficial to the inspection and maintenance of system reliability.

[0043] (6) Comprehensive information capture: Compared with existing radar systems, multi-point positioning systems and other technologies, the present invention can capture the overall appearance information of the aircraft, as well as information on other infrastructure and objects in the flight area, which can be used for aircraft model identification and confirmation, FOD detection, wheel track deviation monitoring, etc.

[0044] (7) Good real-time performance: The feedback process of flight zone monitoring information is short and the information is updated in real time.

[0045] (8) Good system stability: Cameras in different surveillance segments are independent of each other. If a camera fails, other cameras can continue to work and maintain basic working performance in other working segments. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 is a flow chart of a digital mapping method for the operating status of an aircraft on an airport runway in an embodiment,

[0047] Figure 2 A schematic diagram of the location of the image capture device in the embodiment;

[0048] Figure 3 Schematic diagram of the confidence memory structure in the embodiment;

[0049] Figure 4 Schematic diagram of the camera coordinate system and the image coordinate system in the embodiment. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0051] Example 1

[0052] In response to the problems existing in the aforementioned prior art, this embodiment provides a method for acquiring and mapping the operating status of an airport flight zone. Based on the existing machine vision target detection algorithm, after detecting the aircraft in the flight zone, the operating status of the flight zone in the real world, that is, the position, operating speed and other information of the aircraft in the flight zone, is synchronously updated in real time in the digital runway model, so as to display the actual operating status of the flight zone in the digital model, open up the mapping path from the real runway to the digital runway, and thus solve the problems in the prior art.

[0053] See also Figure 1 , the method comprises the following steps:

[0054] S1, collecting real-time images of the flight zone operation through multiple image capture devices, which is implemented by camera sensors in this embodiment.

[0055] S2, input the image data obtained in S1 into the pre-trained target detection model to detect each target in the flight area and obtain the pixel coordinates of the target in the image. Specifically, this step includes:

[0056] S2.1, using the previous and next frame confidence memory structure (such as Figure 3 As shown in the figure, the confidence of the same category of the same grid in the adjacent frame is weighted and transferred to the current frame, which enhances the detection of the current frame by the adjacent frame information. The final confidence calculation method is as follows:

[0057]

[0058] in,

[0059] conf final is the final confidence of the current frame;

[0060] n is the pre-frame memory range, in f;

[0061] p i is the superposition weight of the previous i-th frame;

[0062] conf i is the confidence of the previous i frames.

[0063] S3, according to the installation position of the camera in S1 and the camera parameters, establish the mapping relationship between the coordinates of the target in the image and the real coordinates, and convert the pixel coordinates of the target into the real coordinates. Specifically, this step includes:

[0064] S3.1, convert the image coordinates into real coordinates. According to the pinhole imaging model, the camera coordinate system and the image coordinate system are transformed through the perspective relationship, and similar triangles can be used for calculation to obtain the following relationship:

[0065]

[0066]

[0067] The direct linear transformation relationship between the ideal pixel point and the space point is obtained without considering the distortion and the camera internal parameter deformation, that is, Abdal's direct linear model (DLT). Since the coordinate transformation process in the embodiment is a linear transformation, the calculation method obtained after normalization is as follows:

[0068] X w =x / width*L cap +X c

[0069] in,

[0070] X wis the horizontal coordinate of the world coordinate system, in m;

[0071] x is the horizontal coordinate of the image coordinate system, in px;

[0072] Width is the pixel width of the image, in px;

[0073] L cap is the camera coverage, in meters;

[0074] X c is the camera coordinate, in meters.

[0075] The y-axis position of the aircraft is calculated according to the positional relationship between the nose wheel and the centerline of the runway, and the z-axis coordinate mapping relationship of the aircraft is calibrated according to the perspective relationship based on the installation position and posture of the camera in the embodiment.

[0076] S3.2, when there is a target at the junction of the coverage of two cameras, the coordinates of the targets on the edge of the screen are connected. First, data association is required to determine whether the same target is detected in the two images. Considering that the distance between the aircraft is larger than the size of the detected target aircraft, it is easy to be detected. Based on this prior condition, in the embodiment, a distance threshold (half the length of the fuselage) is set between the two detected targets to make a rough judgment on whether they are the same target.

[0077] Considering that when the target is between two pictures, the two divided parts are theoretically continuous parts, the target coordinates are calculated by a simple averaging method in this embodiment. The simple averaging method of this algorithm is easy to understand and implement, does not require complex algorithms and a large amount of computing resources, can be executed quickly, and is suitable for real-time video surveillance systems. The calculation method is as follows:

[0078] (X3,Y3)=((X1+X2) / 2,(Y1+Y2) / 2)

[0079] in,

[0080] (X1, Y1) is the center coordinate of the detection box in picture 1, px;

[0081] (X2, Y2) is the center coordinate of the detection box in picture 2, px;

[0082] (X3, Y3) is the simple average detection box center coordinate, px.

[0083] S4, the obtained target real coordinates are transmitted to the digital runway, and the real-time position of the target in the digital runway is visualized.

[0084] The method will be described below with a specific example, taking a common aircraft Boeing 737 as an example, the takeoff speed V tThe speed is planned to be 300km / h. It is expected that more than 50 frames of data can be obtained within the coverage range from a single camera position. The shooting frame rate of the camera is planned to be 30fps. The coverage range of a single camera position can be calculated as follows:

[0085] T cap =N exp / FPS=50 / 30=1.67s

[0086] L cap =T cap *V t =1.67*300 / 3.6=139m

[0087] in,

[0088] T cap The time the aircraft is within the camera coverage, in seconds;

[0089] N exp The number of frames expected from a single camera position, in units of f;

[0090] FPS is the shooting frame rate, in f / s;

[0091] L cap is the coverage range of a single camera position, in meters;

[0092] V t is the take-off speed in km / h.

[0093] The coverage of a single camera needs to be no less than 139m. For ease of calculation, the camera spacing is 150m, the camera equipment is installed in a sidelight style, the camera viewing angle is parallel to the ground, and the height from the ground is 30cm. The position diagram is as follows Figure 1 shown.

[0094] Using the method of the present invention, the method for obtaining and mapping the operating status of aircraft in the flight zone is as follows (the data processing process is all performed in Python):

[0095] (1) Obtain aircraft images in the flight zone under various environments at the installation location, annotate them, and train a target detection model (in this embodiment, the yolov5 algorithm is used for model training).

[0096] (2) The confidence memory structure of the previous and next frames is used to transfer the confidence weights of the same category of the same grid in the adjacent frames to the current frame, which enhances the detection of the current frame by the adjacent frame information. This embodiment memorizes the first four frames, and the weights of each frame from the previous to the next are 0.05, 0.1, 0.1, and 0.3 respectively. The final confidence calculation method is as follows:

[0097] conf final=0.05*conf4+0.1*conf3+0.1*conf2+0.3*conf1

[0098] in,

[0099] conf final is the final confidence of the current frame;

[0100] n is the pre-frame memory range, in f;

[0101] p i is the superposition weight of the previous i-th frame;

[0102] conf i is the confidence of the previous i frames.

[0103] (3) According to Abdal's direct linear model (DLT). Since the x-coordinate transformation process in the embodiment is a linear transformation, the calculation method obtained after normalization is as follows:

[0104] X w =x / 1280*150+X c

[0105] in,

[0106] X w is the horizontal coordinate of the world coordinate system, in m;

[0107] x is the horizontal coordinate of the image coordinate system, in px;

[0108] Width is the pixel width of the image, in px;

[0109] X c is the camera coordinate, in meters.

[0110] (4) The y-axis position of the aircraft is calculated based on the positional relationship between the nose wheel and the centerline of the runway, and the z-axis coordinate mapping relationship of the aircraft is calibrated based on the perspective relationship according to the installation position and posture of the camera in the embodiment, and the z-axis coordinate of the aircraft is calculated.

[0111] (5) In the embodiment, the distance threshold between two detected targets is set to 20m to roughly determine whether they are the same target. That is, when the distance between the center coordinates of the two detection frames is less than 20m, it is considered that one target is divided into two parts by the frame.

[0112] Since when the target is between two pictures, the two divided parts are theoretically continuous parts, the target coordinates are calculated by a simple average method in the embodiment. The calculation method is as follows:

[0113] (X3,Y3)=((X1+X2) / 2,(Y1+Y2) / 2)

[0114] in,

[0115] (X1, Y1) is the center coordinate of the detection box in picture 1, px;

[0116] (X2, Y2) is the center coordinate of the detection box in picture 2, px;

[0117] (X3, Y3) is the simple average detection box center coordinate, px.

[0118] (6) The currently obtained coordinates are transferred to the digital runway, and the aircraft model in the digital runway is updated in the following manner:

[0119] The model category is used as the prefix of the model, and the model's x coordinate is used as the suffix of the model name. For example, an airplane target is named plane_x. In this way, the model category and coordinate information can be obtained through the model name, which is convenient for filtering models according to operation needs when traversing the models in the scene.

[0120] Each time an update is made, all models in the current scene are traversed first, and it is determined whether they are aircraft models. A model table is established in the scene to store the model collection that already exists in the scene before the model is added, deleted, or moved. At the same time, a model table to be deleted is established, which is initialized to be empty, to store the collection of models that need to be deleted.

[0121] Then enter the judgment phase to match the model coordinates in the scene with the coordinates in the update coordinate table: if the distance is within a reasonable range, the model position is updated, and the matched update coordinate item is deleted from the update coordinate table to avoid repeated addition of the model; if it is not there, it is marked for deletion (added to the table to be deleted).

[0122] After completing the above steps, the post-processing stage is performed to delete the models in the table to be deleted, and to add models to the remaining unmatched coordinates in the updated coordinate table, name them according to the naming format, and initialize the positions according to the coordinates in the updated coordinate table.

[0123] (7) Visualize the operation status of the flight zone in the digital runway and update the aircraft model in real time in the digital runway to facilitate flight zone monitoring, risk warning and automatic alarm (Unity3d is used as the visualization platform of the digital runway in the embodiment).

[0124] The operation status acquisition method implemented by this method has high positioning accuracy and meets the positioning accuracy requirements for airfield monitoring; the cost is lower than that of traditional monitoring methods, which reduces the construction cost and can be promoted for use in small and medium-sized airports; the mapping data is connected to the digital runway, which is highly scalable and can not only accurately display the real-time operation information of the runway, including the location of the aircraft target, but also provide the real-time operation data required for airfield risk perception and intelligent early warning.

[0125] Example 2

[0126] Based on Example 1, this embodiment provides a method for digitally mapping the aircraft running status of an airport runway as described in Example 1, wherein the digital mapping device includes:

[0127] A plurality of image capture devices are located on the same side of the runway and are arranged at equal intervals along the runway direction and are installed in an edge light style, with the viewing angle of the image capture devices parallel to the ground;

[0128] The data processing terminal is used to obtain images from the image capture device, and realize digital mapping of targets in the flight area through target detection, coordinate conversion and connection processing.

[0129] Example 3

[0130] This embodiment provides an electronic device, including: one or more processors and a memory, wherein the memory stores one or more programs, and the one or more programs include instructions for executing the digital mapping method of the airport runway aircraft operating status as described in Example 1.

[0131] Example 4

[0132] This embodiment provides a computer-readable storage medium, including one or more programs for execution by one or more processors of an electronic device, wherein the one or more programs include instructions for executing the digital mapping method of the airport runway aircraft operating status as described in Example 1.

[0133] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A digital mapping method for the operation status of aircraft on an airport runway, characterized in that: The steps include: Step S1, acquiring multiple images from multiple edge light image capture devices on both sides of the airport runway; Step S2, for each image captured by the image capture device, calculate the confidence of the same target category in the same grid in multiple adjacent frames and weight them as the final confidence of the current frame, obtain the pixel coordinate information of the target in the image, and realize target detection; Step S3, based on the pixel coordinate information of the target in the image, using the pre-constructed image coordinate-real coordinate conversion relationship, converting to obtain real coordinate information, and performing coordinate connection processing on the target located at the intersection of the coverage range of the image capture device; Step S4, the actual coordinate information after coordinate connection processing is transmitted to the pre-built digital runway to realize the digital mapping of the target in the flight area.

2. A digital mapping method for the operation status of an aircraft on an airport runway according to claim 1, characterized in that: In step S2, the final confidence of the current frame is calculated using the following formula: Among them, conf final is the final confidence of the current frame, n is the memory range before the frame, and p i is the superposition weight of the previous i-th frame, conf i is the confidence of the previous i frames.

3. The digital mapping method of the airport runway aircraft operation status according to claim 1 is characterized in that: In step S3, the conversion relationship between image coordinates and real coordinates is constructed as follows: X w =x / width*L cap +X c Among them, X w is the horizontal coordinate of the world coordinate system, x is the horizontal coordinate of the image coordinate system, width is the pixel width of the image, L cap is the image capture device coverage, X c is the camera coordinate.

4. The digital mapping method of the airport runway aircraft operation status according to claim 3 is characterized in that: The image capture device coverage is calculated using the following formula: L cap =T cap *V t Among them, T cap is the time the aircraft is within the coverage of the image capture device, N exp is the number of frames expected by a single image capture device, FPS is the shooting frame rate, L cap Coverage for a single image capture device, V t is the takeoff speed.

5. The digital mapping method of the airport runway aircraft operation status according to claim 1 is characterized in that: In step S3, the process of performing coordinate connection processing on the target located at the boundary of the coverage area of ​​the image capture device includes: Determine whether the distance between the center coordinates of the detection frames of two targets located at the intersection of the coverage range of the image capture device is less than a preset value. If so, merge the two targets into one target and update the coordinates of the target.

6. A digital mapping method for the operation status of an aircraft on an airport runway according to claim 5, characterized in that: The above-mentioned merging of two targets into one target and updating the coordinates of the target is implemented by the following formula: (X3,Y3)=((X1+X2) / 2,(Y1+Y2) / 2) Among them, (X1, Y1) and (X2, Y2) are the center coordinates of the detection box of the two targets before merging, and (X3, Y3) are the center coordinates of the detection box of the updated target.

7. The digital mapping method of the airport runway aircraft operation status according to claim 1 is characterized in that: The step S4 comprises: Step S401, matching the real coordinate information after the coordinate connection processing with the coordinates of the model in the scene at the previous moment, and judging whether the difference between the real coordinates and the coordinates of the model in the scene is less than a preset value, if so, updating the coordinates of the model in the scene to the real coordinates, if not, deleting the model in the scene; Step S402, adding a model to the scene for the coordinates that are not matched in the real coordinate information; Step S403: output the real-time operation information of the runway.

8. A digital mapping device for the operation status of aircraft on an airport runway, characterized in that: A method for digitally mapping the operation status of an aircraft on an airport runway as claimed in any one of claims 1 to 7, wherein the digital mapping device comprises: A plurality of image capture devices are located on the same side of the runway and are arranged at equal intervals along the runway direction and are installed in an edge light style, with the viewing angle of the image capture devices parallel to the ground; The data processing terminal is used to obtain images from the image capture device, and realize digital mapping of targets in the flight area through target detection, coordinate conversion and connection processing.

9. An electronic device, characterized in that: include: One or more processors and a memory, wherein the memory stores one or more programs, and the one or more programs include instructions for executing the digital mapping method of the airport runway aircraft operating status as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: It comprises one or more programs for execution by one or more processors of an electronic device, wherein the one or more programs comprise instructions for executing the digital mapping method of the airport runway aircraft operating status as described in any one of claims 1-7.

Images