Intelligent monitoring method, device and system for flight support nodes

Abstract

The present invention discloses an intelligent monitoring method, device and system for flight support nodes, which can realize the intelligent collection of the completion status of each flight support node, reduce the delay error caused by manual input, thereby improving the efficiency and accuracy of flight support, and reducing the consumption of manpower and material resources. The method comprises: using a target detection model based on a yolov5 network to intelligently identify video frames at different times of a multi-view video stream corresponding to any aircraft position in an airport apron, so as to determine the first flight support node currently executed by any aircraft position, and update the flight support node of any aircraft position to the first flight support node; further determine and based on the detection category to which the first flight support node belongs, use a preset method to monitor the operation process status of the first flight support node, and record the operation time of the first flight support node; repeat the above steps until it is determined that the first flight support node is the last flight support node.

Description

Technical Field

[0001] The present invention relates to the technical field of airport flight support operations, and in particular to an intelligent monitoring method, device and system for flight support nodes. Background Art

[0002] With the rapid development of the civil aviation industry, both passenger and cargo volumes are increasing rapidly, which brings huge development space to the civil aviation industry, but also puts forward higher requirements for the business efficiency of civil aviation. However, the flight support capabilities of major civil aviation airports are not satisfactory, and flight delays caused by flight support efficiency issues are still widespread. Therefore, it is urgent to realize intelligent perception of the flight support process.

[0003] Flight support refers to a series of support operations performed after the aircraft approaches the airport, including wheel chocks, aircraft refueling, food delivery, and luggage assembly. Currently, the flight support operation processes of major domestic airports require manual supervision, and ground staff are required to manually confirm the progress of the support operations and complete the confirmation of key nodes of the support operations. This operation method not only updates information slowly, but is also limited by the work experience and work status of the ground staff, and there may be errors or even mistakes in the support information. The traditional manual operation method not only consumes a lot of manpower and material resources, but the support information obtained may also be inaccurate, which seriously affects the airport's scheduling of flight support and the handling of abnormal situations. Summary of the invention

[0004] Based on this, the purpose of the present invention is to provide an intelligent monitoring method, device and system for flight support nodes, which is used to improve the efficiency and accuracy of flight support, reduce the consumption of manpower and material resources, and help airports to schedule flight support and handle abnormal situations.

[0005] In a first aspect, the present invention provides an intelligent monitoring method for a flight support node, comprising:

[0006] Get the multi-view video stream corresponding to any aircraft position in the airport apron;

[0007] Using a target detection model based on the yolov5 network to intelligently identify video frames at different times of the multi-view video stream, and determine the status information of the target aircraft and target objects located in any of the aircraft positions; the target objects include one or more of various types of support vehicles, aircraft wheels, cargo doors and cabin doors;

[0008] Based on the status information of the target aircraft and the target object and the flight support node operation process, determine the first flight support node currently executed by any one of the aircraft stands, and update the flight support node of any one of the aircraft stands to the first flight support node; the flight support node operation process includes the time sequence relationship between each flight support node;

[0009] Determine the detection category to which the first flight support node belongs, and based on the detection category, monitor the operation process status of the first flight support node in a preset manner, and record the operation time of the first flight support node;

[0010] If it is determined that the first flight support node is the last flight support node in the flight support node operation process, it is determined that all flight support nodes corresponding to any one of the aircraft stands have been monitored; or

[0011] If it is determined that the first flight support node is not the last flight support node in the flight support node operation process, return to execute to obtain the multi-view video stream corresponding to any aircraft position in the airport apron, and continue to monitor the flight support node of any aircraft position.

[0012] In a possible design, a multi-view video stream corresponding to any aircraft stand in an airport apron is obtained, including:

[0013] The multi-view video stream transmitted by the streaming media server is obtained; the multi-view video stream is collected by a plurality of video acquisition terminals monitoring different viewing angles of any camera position and is saved in real time to the streaming media server.

[0014] In a possible design, the target detection model based on the yolov5 network adopts the yolov5 network as the basic structure, uses the nearest neighbor association algorithm and the adjacent frame difference method to judge the motion state of the target aircraft and the target object; the target detection model based on the yolov5 network is used to intelligently identify the video frames at different times of the multi-view video stream to determine the state information of the target aircraft and the target object located in any of the camera positions, including:

[0015] The yolov5 network of the target detection model based on the yolov5 network is used to identify the video frames of the multi-view video stream at different times, and the positions of the target aircraft and the target object in the video frames at different times are output;

[0016] Based on the positions of the target aircraft and the target object in the video frames at different times, the nearest neighbor association algorithm and adjacent frame difference method of the target detection model based on the yolov5 network are used to determine the position change of the target aircraft and the target object between the video frames at different times;

[0017] Based on the position changes of the target aircraft and the target object between the video frames at different times, the state information of the target aircraft and the target object located in any one of the aircraft positions is determined.

[0018] In a possible design, the flight support node operation process further includes a detection category of each flight support node, where the detection category is one of a target detection category, a pedestrian recognition and direction judgment category, and a motion detection category; determining the detection category of the first flight support node, and based on the detection category, monitoring the operation process status of the first flight support node in a preset manner, including:

[0019] Determining a detection category of the first flight support node based on the flight support node operation process;

[0020] If the detection category of the first flight support node is the target detection category, based on the status information of the target aircraft and the target object, the docking relationship between the target aircraft and the target object is monitored; based on the docking relationship, the operation process status of the first flight support node is monitored;

[0021] Alternatively, if the detection category of the first flight support node is the pedestrian recognition and direction judgment category, the target object includes the passenger elevator car, and the yolov5 network based on the target detection model of the yolov5 network adaptively generates optical flow calculation areas and pedestrian detection areas in the video frames of the multi-view video stream at different times; performs optical flow calculation on the optical flow calculation area to determine the up and down status of the pedestrians on the passenger elevator car, and performs pedestrian identity recognition on the pedestrian detection area to determine the identity attributes of the pedestrians on the passenger elevator car; based on the up and down status of the pedestrians and the pedestrian identity attributes, monitors the operation process status of the first flight support node;

[0022] Alternatively, if the detection category of the first flight support node is the action detection category, the yolov5 network based on the target detection model of the yolov5 network adaptively generates action detection areas in the video frames of the multi-view video stream at different times; personnel support operation action detection is performed on the action detection area to determine the current support operation action of the personnel in the first flight support node; based on the current support operation action, the operation process status of the first flight support node is monitored.

[0023] In a possible design, the optical flow calculation area is monitored for optical flow to determine the up and down state of the pedestrians on the passenger elevator, including: generating a corner point distribution area of ​​the optical flow calculation area based on the Shi-Tomasi algorithm; the corner point distribution area is a rectangular area formed by the 1 / 5 positions on both sides of the upper left corner and the 1 / 5 positions on both sides of the lower right corner of the passenger elevator; using the optical flow algorithm to perform optical flow calculation on the corner point distribution area to obtain a sparse optical flow of the optical flow calculation area; if the vertical movement of the optical flow in the optical flow calculation area is determined to be upward based on the sparse optical flow, then the up and down state of the pedestrians on the passenger elevator is determined to be upward, or, if the vertical movement of the optical flow in the optical flow calculation area is determined to be downward based on the sparse optical flow, then the up and down state of the pedestrians on the passenger elevator is determined to be downward;

[0024] Performing pedestrian identification in the pedestrian detection area to determine the identity attributes of pedestrians on the passenger elevator, including: performing pedestrian identification in the pedestrian detection area based on a target detection model of the EfficientDet algorithm, using a PCBReID network to extract a unified feature vector of pedestrians in the pedestrian detection area, and calculating the maximum cosine similarity between the unified feature vectors of pedestrians; and determining the pedestrian identity attributes of people on the passenger elevator based on the maximum cosine similarity.

[0025] In a possible design, performing personnel support operation action detection on the action detection area to determine the current support operation action of the personnel in the first flight support node includes:

[0026] A dual-stream based action recognition network is used to detect personnel support operation actions in the action detection area, and the confidence of the support operation actions occurring in the video frames of the multi-view video stream at different times is output; the support operation action corresponding to the maximum confidence is determined as the current support operation action of the personnel in the first flight support node.

[0027] In a possible design, after determining the detection category to which the first flight support node belongs, and based on the detection category, monitoring the operation process status of the first flight support node in a preset manner, and recording the operation time of the first flight support node, the method further includes:

[0028] Obtaining a code of any of the camera positions;

[0029] Generate structured data based on the operation process status and operation time of the first flight support node and the code; the structured data uses the number of any one of the aircraft positions as a primary key and saves the operation process status and operation time of the first flight support node;

[0030] The structured data is stored in a relational database.

[0031] In one possible design, the method further includes:

[0032] The multi-view video stream is played through a video platform, and the operation process status and operation time of the first flight support node are displayed.

[0033] In a second aspect, the present invention further provides an intelligent monitoring device for a flight support node, comprising:

[0034] A target detection unit is used to obtain a multi-view video stream corresponding to any aircraft position in an airport apron; a target detection model based on a yolov5 network is used to intelligently identify video frames at different times of the multi-view video stream, and determine the status information of a target aircraft and a target object located in any aircraft position; the target object includes one or more of various types of support vehicles, aircraft wheels, cargo doors, and cabin doors; based on the status information of the target aircraft and the target object and the flight support node operation process, determine the first flight support node currently executed by any aircraft position, and update the flight support node of any aircraft position to the first flight support node; the flight support node operation process includes the timing relationship between each flight support node;

[0035] A monitoring unit is used to determine the detection category to which the first flight support node belongs, and based on the detection category, monitor the operation process status of the first flight support node in a preset manner, and record the operation time of the first flight support node; if it is determined that the first flight support node is the last flight support node in the operation process of the flight support node, determine that the monitoring of all flight support nodes corresponding to any aircraft stand has been completed; or, if it is determined that the first flight support node is not the last flight support node in the operation process of the flight support node, return to obtain the multi-view video stream corresponding to any aircraft stand in the airport apron through the target detection unit, and continue to monitor the flight support node of any aircraft stand.

[0036] In a possible design, the target detection unit is specifically used to:

[0037] The multi-view video stream transmitted by the streaming media server is obtained; the multi-view video stream is collected by a plurality of video acquisition terminals monitoring different viewing angles of any camera position and is saved in real time to the streaming media server.

[0038] In a possible design, the target detection model based on the yolov5 network adopts the yolov5 network as the basic structure, and uses the nearest neighbor association algorithm and the adjacent frame difference method to judge the motion state of the target aircraft and the target object; the target detection unit is specifically used for:

[0039] The yolov5 network of the target detection model based on the yolov5 network is used to identify the video frames of the multi-view video stream at different times, and the positions of the target aircraft and the target object in the video frames at different times are output;

[0040] Based on the positions of the target aircraft and the target object in the video frames at different times, the nearest neighbor association algorithm and adjacent frame difference method of the target detection model based on the yolov5 network are used to determine the position change of the target aircraft and the target object between the video frames at different times;

[0041] Based on the position changes of the target aircraft and the target object between the video frames at different times, the state information of the target aircraft and the target object located in any one of the aircraft positions is determined.

[0042] In a possible design, the flight support node operation process also includes the detection category of each flight support node, and the detection category is one of target detection, pedestrian recognition and direction judgment, and motion detection; the monitoring unit includes an optical flow calculation module, a pedestrian attribute analysis module, and a motion detection module; the monitoring unit is specifically used to:

[0043] Determining a detection category of the first flight support node based on the flight support node operation process;

[0044] If the detection category of the first flight support node is the target detection category, based on the status information of the target aircraft and the target object from the target detection unit, monitor the docking relationship between the target aircraft and the target object; based on the docking relationship, monitor the operation process status of the first flight support node;

[0045] Alternatively, if the detection category of the first flight support node is the pedestrian recognition and direction judgment category, the target object includes a passenger elevator car, and the optical flow calculation area and the pedestrian detection area in the video frames of the multi-view video stream at different times are adaptively generated through the yolov5 network of the target detection model based on the yolov5 network of the target detection unit; the optical flow calculation module is used to perform optical flow calculation on the optical flow calculation area to determine the up and down status of the pedestrians on the passenger elevator car, and the pedestrian attribute analysis module is used to perform pedestrian identity recognition on the pedestrian detection area to determine the identity attributes of the pedestrians on the passenger elevator car; based on the up and down status of the pedestrians and the pedestrian identity attributes, the operation process status of the first flight support node is monitored;

[0046] Alternatively, if the detection category of the first flight support node is the action detection category, the action detection area in the video frames of the multi-view video stream at different times is adaptively generated through the yolov5 network of the target detection model based on the yolov5 network of the target detection unit; the action detection module is used to perform personnel support operation action detection on the action detection area to determine the current support operation action of the personnel in the first flight support node; based on the current support operation action, the operation process status of the first flight support node is monitored.

[0047] In a possible design, the optical flow calculation module is specifically used to: generate a corner point distribution area of ​​the optical flow calculation area based on the Shi-Tomasi algorithm; the corner point distribution area is a rectangular area formed by the 1 / 5 positions on both sides of the upper left corner and the 1 / 5 positions on both sides of the lower right corner of the passenger elevator; use the optical flow algorithm to perform optical flow calculation on the corner point distribution area to obtain a sparse optical flow of the optical flow calculation area; if it is determined based on the sparse optical flow that the vertical direction of the optical flow in the optical flow calculation area is upward, then the up and down state of the pedestrian on the passenger elevator is determined to be upward, or, if it is determined based on the sparse optical flow that the vertical direction of the optical flow in the optical flow calculation area is downward, then the up and down state of the pedestrian on the passenger elevator is determined to be downward;

[0048] The pedestrian attribute analysis module is specifically used to: perform pedestrian recognition in the pedestrian detection area based on the target detection model of the EfficientDet algorithm, use the PCB ReID network to extract the unified feature vector of pedestrians in the pedestrian detection area, and calculate the maximum cosine similarity between the unified feature vectors of pedestrians; based on the maximum cosine similarity, determine the pedestrian identity attributes of the people on the passenger elevator.

[0049] In a possible design, the action detection module is specifically used to:

[0050] A dual-stream based action recognition network is used to detect personnel support operation actions in the action detection area, and the confidence of the support operation actions occurring in the video frames of the multi-view video stream at different times is output; the support operation action corresponding to the maximum confidence is determined as the current support operation action of the personnel in the first flight support node.

[0051] In a possible design, the monitoring unit is further used for:

[0052] Obtaining a code of any of the camera positions;

[0053] Generate structured data based on the operation process status and operation time of the first flight support node and the code; the structured data uses the number of any one of the aircraft positions as a primary key and saves the operation process status and operation time of the first flight support node;

[0054] The structured data is stored in a relational database.

[0055] In a possible design, the monitoring unit is further used for:

[0056] The multi-view video stream is played through a video platform, and the operation process status and operation time of the first flight support node are displayed.

[0057] In a third aspect, the present invention further provides an intelligent monitoring device for a flight support node, comprising: at least one memory and at least one processor;

[0058] The at least one memory is used to store one or more programs;

[0059] When the one or more programs are executed by the at least one processor, the method involved in any possible design of the first aspect described above is implemented.

[0060] In a fourth aspect, the present invention further provides an intelligent monitoring system for a flight support node, comprising: a plurality of video acquisition terminals located at different viewing angles of any aircraft position, a streaming media server, an intelligent monitoring device for a flight support node, a relational database and a video platform, wherein the streaming media server is respectively connected to the video acquisition terminal and the intelligent monitoring device for the flight support node, and the relational database is respectively connected to the intelligent monitoring device for the flight support node and the video platform; wherein,

[0061] The video acquisition terminal is used to acquire the video stream of the corresponding viewing angle of any camera position, and transmit the acquired video stream to the streaming media server in real time;

[0062] The streaming media server is used to store the multi-viewing angle video stream of any camera position;

[0063] The intelligent monitoring device of the flight support node is used to implement the method involved in any possible design of the first aspect above;

[0064] The relational database is used to obtain the operation process status and operation time of the first flight support node from the intelligent monitoring device of the flight support node, and obtain the code of any one of the aircraft positions, generate structured data based on the operation process status and operation time of the first flight support node and the code, and save the structured data;

[0065] The video platform is used to play the multi-view video stream and display the operation process status and operation time of the first flight support node.

[0066] In a fifth aspect, the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores at least one program; when the at least one program is executed by a processor, the method involved in any possible design of the above-mentioned first aspect is implemented.

[0067] The beneficial effects of the present invention are as follows:

[0068] In the technical solution provided by the present invention, a multi-view video stream corresponding to any aircraft stand in an airport apron is obtained, and a target detection model based on a yolov5 network is used to intelligently identify video frames at different times of the multi-view video stream, and the status information of a target aircraft and a target object located in any aircraft stand is determined; based on the status information of the target aircraft and the target object and the flight support node operation process, the first flight support node currently executed by any aircraft stand is determined, and the flight support node of any aircraft stand is updated to the first flight support node; the detection category to which the first flight support node belongs is determined, and based on the detection category, the operation process status of the first flight support node is monitored in a preset manner, and the operation time of the first flight support node is recorded; if it is determined that the first flight support node is the last flight support node in the flight support node operation process, it is determined that all flight support nodes corresponding to the any aircraft stand have been monitored; or, if it is determined that the first flight support node is not the last flight support node in the flight support node operation process, the multi-view video stream corresponding to any aircraft stand in the airport apron is returned to be executed to obtain, and the flight support node of any aircraft stand is continued to be monitored. In this way, the operation process status and operation time of all flight support nodes at any aircraft stand can be monitored automatically and intelligently without human intervention, thereby realizing the intelligent collection of the completion status of each node of flight support. Compared with the traditional manual operation method, it can avoid the interference of human factors and reduce the delay error caused by manual entry, thereby improving the accuracy of the obtained support information. At the same time, it can also reduce the consumption of manpower and material resources in the operation of aviation support nodes, thereby improving the efficiency and accuracy of flight support, reducing the consumption of manpower and material resources, and helping the airport to dispatch flight support and handle abnormal situations.

[0069] For better understanding and implementation, the present invention is described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 A schematic diagram of a flow chart of an intelligent monitoring method for a flight support node provided by the present invention;

[0071] Figure 2 A schematic diagram of a flight support node operation process provided by the present invention;

[0072] Figure 3 A schematic diagram of a corner point distribution area provided by the present invention;

[0073] Figure 4 A flow chart of another intelligent monitoring method for flight support nodes provided by the present invention;

[0074] Figure 5 A structural schematic diagram of an intelligent monitoring device for a flight support node provided by the present invention;

[0075] Figure 6 A structural schematic diagram of another intelligent monitoring device for a flight support node provided by the present invention;

[0076] Figure 7 A schematic diagram of the architecture of an intelligent monitoring system for flight support nodes provided by the present invention. DETAILED DESCRIPTION

[0077] The directional terms such as up, down, left, right, front, back, front, back, top, bottom, etc. mentioned or may be mentioned in this specification are defined relative to their structure and are relative concepts. Therefore, they may change accordingly according to their different locations and different usage conditions. Therefore, these or other directional terms should not be interpreted as restrictive terms.

[0078] The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of embodiments consistent with some aspects of the present disclosure.

[0079] The terms used in this disclosure are only for the purpose of describing specific embodiments and are not intended to limit the disclosure. The singular forms of "a", "the" and "the" used in this disclosure are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in this disclosure refers to and includes any or all possible combinations of one or more associated listed items.

[0080] Unless otherwise specified, ordinal numbers such as "first" and "second" mentioned in the present disclosure are used to distinguish multiple objects and are not used to limit the order, timing, priority or importance of the multiple objects.

[0081] Please refer to Figure 1 As shown, the intelligent monitoring method of the flight support node provided by the present invention comprises the following steps:

[0082] S11. Obtain a multi-view video stream corresponding to any aircraft position in the airport apron.

[0083] In some embodiments, a multi-view video stream corresponding to any camera position transmitted by a streaming media server may be obtained, wherein the multi-view video stream corresponding to any camera position may be collected by a plurality of video acquisition terminals monitoring different viewing angles of any camera position and saved in real time to the streaming media server.

[0084] Different support operations generally occur at different locations of the aircraft. In order to maximize the monitoring of multiple flight support nodes, multi-visual methods can be used for monitoring during specific implementation. For example, for any aircraft stand, such as a remote aircraft stand apron, three video acquisition terminals can be deployed to cover the monitoring, such as deploying three video acquisition terminals to respectively aim at the left, middle and right positions of the aircraft nose.

[0085] Exemplarily, the heights of the three video acquisition terminals can be set slightly higher than the aircraft, and the left and right angles of the two video acquisition terminals aimed at the left and right positions of the aircraft nose can be set to about 30 degrees. It can be understood that the angle between the two-point connecting line between the two video acquisition terminals and the aircraft nose and the forward straight line of the aircraft nose is about 30 degrees.

[0086] In specific implementation, each video acquisition terminal can save the collected video stream in real time to the streaming media server. For example, each video acquisition terminal can use the Real Time Streaming Protocol (RTSP) to communicate and transmit data with the streaming media server, and use the H264 format to save the video stream in real time to the streaming media server.

[0087] In the present invention, by adopting different viewing angles to monitor any aircraft position, it is convenient to subsequently monitor the same flight support node from different viewing angles, and when the monitoring results are integrated, monitoring with better robustness can be achieved.

[0088] In specific implementation, the streaming media server can parse, transcode and store the video streams from each video acquisition terminal as video data. Afterwards, when the streaming media server is requested to obtain the video data, the streaming media server transmits the video data in the form of a video stream. The streaming media server can also use the Real Time Messaging Protocol (RTMP) to play the stored video data, and provide the stored video data to other monitoring systems or video systems.

[0089] S12. Use the target detection model based on the yolov5 network to intelligently identify the video frames of the multi-view video stream at different times, and determine the status information of the target aircraft and target objects located in any position.

[0090] In some embodiments, the target object may include, but is not limited to, one or more of various support vehicles, aircraft wheels, cargo doors, and cabin doors. Among them, various support vehicles may include, but are not limited to, passenger elevators, fuel trucks, trucks, conveyor belt trucks, connecting rod carts, wheel carts, food trucks, passenger shuttles, and garbage trucks.

[0091] In the specific implementation, before executing step S12, the target detection model based on the yolov5 network can be trained on a data set of a specific remote parking apron perspective, wherein the target detection data set needs to locate and annotate each target object to be identified with a rectangular frame and assign a category label. The category label consists of aircraft, various support vehicles, aircraft wheels, cargo doors, and cabin doors. A data set is constructed and divided into a training set, a validation set, and a test set.

[0092] In a specific implementation, before executing step S12, after obtaining the multi-view video stream corresponding to any camera position, the multi-view video stream can be subjected to frame extraction processing to save the computing power and optimization speed of the target detection model based on the yolov5 network.

[0093] In the specific implementation, before executing step S12, since the acquisition area of ​​the video acquisition terminal of any camera position may cover the areas of other camera positions, in order for the target detection model based on the yolov5 network to focus on intelligent identification of any camera position and improve the calculation speed, the multi-view video stream corresponding to any camera position can be processed by frame extraction, and a mask can be added to the video stream corresponding to any camera position.

[0094] In some embodiments, the target detection model based on the yolov5 network can use the yolov5 network as the basic structure, and use the nearest neighbor association algorithm and the adjacent frame difference method to judge the motion state of the target aircraft and the target object.

[0095] For example, the target detection model based on the yolov5 network can use the yolov5 network as the basic structure, the maximum number of detected targets is set to 1000, the non-maximum suppression threshold is set to 0.45, the confidence threshold is set to 0.25, and the three-way reasoning is merged during reasoning, that is, batch_size is 3, to reduce the reasoning time. In terms of motion state judgment, the target detection model based on the yolov5 network can use the nearest neighbor association algorithm and the adjacent frame difference method, match the target relationship of adjacent frames based on the nearest neighbor principle, and judge the state of motion and stillness based on the position and detection box area. When the distance between the targets is less than the docking threshold and the state is still, it is judged as docking; when it is greater than the separation threshold and the state is moving, it is judged as separation. In order to improve the robustness of the target detection model based on the yolov5 network, the target detection model based on the yolov5 network can use Kalman filtering to obtain stable detection.

[0096] In the specific implementation, in step S12, the yolov5 network of the target detection model based on the yolov5 network can be used to identify the video frames of the multi-view video stream corresponding to any camera position at different times, and the positions of the target aircraft and the target object in the video frames at different times are output. Afterwards, based on the positions of the target aircraft and the target object in the video frames at different times, the nearest neighbor association algorithm and the adjacent frame difference method of the target detection model based on the yolov5 network are used to determine the position change of the target aircraft and the target object between the video frames at different times. Based on the position change of the target aircraft and the target object between the video frames at different times, the state information of the target aircraft and the target object located in any camera position can be determined.

[0097] For example, taking the target aircraft as an example, the position of the target aircraft in the i-th video frame can be matched with the position of the target aircraft in the i-1th video frame. The matching principle is to minimize the distance from the center point of the previous video frame. In this way, the same target aircraft is tracked. Among them, 2≦i≦N+1, i is rounded, and N is equal to the total number of video frames at different times. When the detection box area of ​​the target aircraft does not change by more than the set threshold in multiple consecutive video frames, and the center point coordinate position change does not change by more than the set threshold, it can be determined that the state of the target aircraft is stationary, otherwise it is determined that the state of the target aircraft is moving.

[0098] S13: Based on the status information of the target aircraft and the target object and the flight support node operation process, determine the first flight support node currently executed by any aircraft stand, and update the flight support node of any aircraft stand to the first flight support node.

[0099] In some embodiments, the flight support node operation process may include the timing relationship between each flight support node. Figure 2 As shown, the flight support node process may include, but is not limited to, the timing relationship between 24 flight support nodes, including aircraft positioning, wheel block loading, refueling start, refueling end, catering start, catering end, wheel block removal, and aircraft pushout.

[0100] As an example, the state information of the target aircraft may include a moving state and a stationary state. If the target object is one of the support vehicles, the state information of the target object may include a stationary state, a moving state toward the target aircraft, and a moving state away from the target aircraft; or, if the target object is one of the cargo door and the cabin door, the state information of the target object may include a closed stationary state, a closed door moving state, a door opening prohibited state, and a door opening moving state; or, if the target object is an aircraft wheel, the state information of the target object may include a prohibited state and a moving state.

[0101] For example, taking the case where the target object appearing in any camera position includes a vehicle as an example, refer to Figure 2 As shown, when the state information of the target aircraft is stationary and the state information of the passenger elevator is moving towards the target aircraft, the first flight support node currently executed by any aircraft stand can be determined as the passenger elevator flight support node based on the flight support node process.

[0102] In specific implementation, after determining the first flight support node currently executed by any aircraft stand, the flight support node of any aircraft stand can be updated to the first flight support node, so that the flight support node of any aircraft stand can be updated in time.

[0103] S14: Determine the detection category to which the first flight support node belongs, and based on the detection category of the first flight support node, monitor the operation process status of the first flight support node in a preset manner, and record the operation time of the first flight support node.

[0104] In some embodiments, the flight support node operation process may further include a detection category of each flight support node, where the detection category is one of a target detection category, a pedestrian recognition and direction judgment category, and a motion detection category.

[0105] refer to Figure 2 As shown, for example, the flight support nodes such as the aircraft entering the position and the passenger elevator only need target detection to realize node status monitoring, so the detection category of the flight support nodes such as the aircraft entering the position and the passenger elevator can be pre-set as the target detection category. For example, the flight support nodes such as the passenger disembarking and the start of cleaning need to combine target detection and personnel movement direction and identity attributes to realize node status detection, so the detection category of the flight support nodes such as the passenger disembarking and the start of cleaning is pre-set as the pedestrian recognition and direction judgment category. For example, the upper wheel gear and lower wheel gear need to combine target detection and support personnel action detection to realize node status detection, so the detection category of the flight support nodes such as the upper wheel gear and lower wheel gear is pre-set as the action detection category.

[0106] In some embodiments, the detection category of the first flight support node can be determined based on the flight support node operation process. Afterwards, based on the detection category of the first flight support node, the operation process status of the first flight support node can be monitored in a preset manner to achieve intelligent and automatic detection of the node status of the flight support node of any aircraft position. In this process, the operation time of the first flight support node can be recorded to achieve overtime operation warning for the first flight support node.

[0107] In some embodiments, if the detection category of the first flight support node is target detection, the docking relationship between the target aircraft and the target object can be monitored and determined based on the status information of the target aircraft and the target object; and the operation process status of the first flight support node can be monitored based on the docking relationship.

[0108] For example, taking the first flight support node as the passenger elevator flight support node, refer to Figure 2 As shown, when the state information of the target aircraft is in a stationary state and the state information of the passenger elevator is in a stationary state, the docking relationship between the target aircraft and the passenger elevator can be monitored, and the docking relationship can include undocking and docking. After determining that the target aircraft and the passenger elevator have not completed docking based on the docking relationship, the operation process status of the first flight support node can be determined to be in execution, or after determining that the target aircraft and the passenger elevator have completed docking based on the docking relationship, the operation process status of the first flight support node can be determined to be completed.

[0109] In some embodiments, if the detection category of the first flight support node is pedestrian recognition and direction judgment, it means that the target object includes the object car, then the yolov5 network based on the target detection model of the yolov5 network can adaptively generate the optical flow calculation area and pedestrian detection area in the video frames of the multi-view video stream at different times. Among them, the optical flow calculation area can be the up and down area of ​​the passenger elevator, and the pedestrian detection area can be the area near the exit of the passenger elevator. Afterwards, the optical flow calculation area can be performed to determine the up and down status of the pedestrians on the passenger elevator, and the pedestrian identity recognition can be performed on the pedestrian detection area to determine the identity attributes of the pedestrians on the passenger elevator; based on the up and down status of the pedestrians and the pedestrian identity attributes, the operation process status of the first flight support node is monitored.

[0110] In specific implementation, the corner point distribution area of ​​the optical flow calculation area can be generated based on the Shi-Tomasi algorithm to generate a sufficient number of corner points to calculate the sparse optical flow of the optical flow calculation area. The corner point distribution area is a rectangular area formed by the 1 / 5 position on both sides of the upper left corner and the 1 / 5 position on both sides of the lower right corner of the passenger elevator. As an example, the maximum number of corner points can be set to 50 and the minimum spacing can be set to 3px. Figure 3 is an example of a corner point distribution area. Afterwards, an optical flow algorithm can be used to perform optical flow calculation on the corner point distribution area to obtain a sparse optical flow in the optical flow calculation area. If the vertical movement of the optical flow in the optical flow calculation area is determined to be upward based on the sparse optical flow, then the up and down state of the pedestrians on the passenger elevator in any position is determined to be upward, or if the vertical movement of the optical flow in the optical flow calculation area is determined to be downward based on the sparse optical flow, then the up and down state of the pedestrians on the passenger elevator is determined to be downward.

[0111] For example, after initializing the corner points of the optical flow calculation area of ​​the video frames of the multi-view video stream at different times, the movement of the corner points of the optical flow calculation area of ​​the video frames of the multi-view video stream at different times is calculated frame by frame. When the average movement direction is upward and greater than the set threshold, it is judged as upward movement, otherwise it is downward movement.

[0112] Since the pedestrian area is relatively small under the monitoring angle of any camera position, it is difficult to use a classifier to classify the pedestrian identity. In specific implementation, the pedestrian detection area can be identified based on the target detection model of the EfficientDet algorithm, and the unified feature vector of pedestrians in the pedestrian detection area can be extracted using the PCB ReID network. The maximum cosine similarity between the unified feature vectors of pedestrians is calculated; based on the maximum cosine similarity, the pedestrian identity attributes of the people on the passenger elevator are determined. For example, the cosine similarity range corresponding to the pedestrian identity attribute, after determining the maximum cosine similarity between the unified feature vectors of pedestrians, the cosine similarity range corresponding to the maximum cosine similarity is determined, so as to determine the pedestrian identity attributes of the people on the passenger elevator.

[0113] In specific implementation, the pedestrian identity attributes may include, but are not limited to: cleaning staff, crew members, and passengers.

[0114] In specific implementation, taking the first flight support node as the passenger disembarkation flight support node as an example, if it is determined that the passenger is in a downward movement state, then the operation process state of the first flight support node can be determined to be in execution. Alternatively, if it is determined that the passenger's downward movement state has changed to a state where there are no passengers on the passenger elevator, then the operation process state of the first flight support node can be determined to be completed.

[0115] In some embodiments, if the detection category of the first flight support node is action detection, the yolov5 network based on the target detection model of the yolov5 network adaptively generates action detection areas in the video frames of the multi-view video stream at different times. Among them, the action detection area may include the wheel area and / or the aircraft area. Afterwards, the action detection area may be subjected to personnel support operation action detection to determine the support operation action of the personnel in the action detection area; based on the support operation action, the operation process status of the first flight support node is monitored.

[0116] In specific implementation, a dual-stream based action recognition network can be used to detect personnel support operation actions in the action detection area. The action information and image semantic information can be integrated to predict the time period when the corresponding support operation action may occur at the first flight support node, and the confidence of the support operation action occurring in the video frames at different times of the multi-view video stream can be output; the support operation action corresponding to the maximum confidence is determined as the current support operation action of the personnel in the first flight support node.

[0117] As an example, the input of the dual-stream action recognition network can be set to a fixed 384 frames, and the input can be updated every 32 frames for reasoning. During reasoning, the I3D network is used for feature extraction and analysis, and the video frames at different times of the multi-view video stream are output to ensure the confidence of the operation action.

[0118] In specific implementation, the operation process status of the first flight support node can be monitored based on the current support operation action of the personnel of the first flight support node. For example, taking the first flight support node as the upper wheel block flight support node as an example, when the current support operation action of the personnel in the first flight support node is the operation of holding the wheel block, then the operation process status of the first flight support node can be determined to be in progress; or, when the current support operation action of the personnel in the first flight support node is the operation of placing the wheel block under the aircraft wheel, then the operation process status of the first flight support node can be determined to be completed.

[0119] In the present invention, based on the detection category of the first flight support node, a preset method is adopted to monitor the operation process status of the first flight support node and record the operation time of the first flight support node. This can realize automatic and intelligent monitoring of all flight support nodes at any aircraft stand without human intervention.

[0120] S15, judging whether the first flight support node is the last flight support node in the flight support node operation flow. If the first flight support node is the last flight support node in the flight support node operation flow, executing step S16, otherwise executing step S17.

[0121] S16, determining that all flight support nodes corresponding to any aircraft stand have been monitored.

[0122] After step S16, the same method can be used to monitor the next round of flight support node operation process for any aircraft stand.

[0123] S17, returning to step S11, and continuing to monitor the flight support node of any aircraft stand.

[0124] In the present invention, a multi-view video stream corresponding to any aircraft position in an airport apron is obtained, and a target detection model based on a yolov5 network is used to intelligently identify video frames of the multi-view video stream at different times, and the status information of a target aircraft and a target object located in any aircraft position is determined; based on the status information of the target aircraft and the target object and the flight support node operation process, the first flight support node currently executed by any aircraft position is determined, and the flight support node of any aircraft position is updated to the first flight support node; the detection category to which the first flight support node belongs is determined, and based on the detection category, the operation process status of the first flight support node is monitored in a preset manner, and the operation time of the first flight support node is recorded; if it is determined that the first flight support node is the last flight support node in the flight support node operation process, it is determined that all flight support nodes corresponding to the any aircraft position have been monitored; or, if it is determined that the first flight support node is not the last flight support node in the flight support node operation process, the multi-view video stream corresponding to any aircraft position in the airport apron is returned to be executed to obtain, and the flight support node of any aircraft position is continued to be monitored. In this way, the operation process status and operation time of all flight support nodes at any aircraft stand can be monitored automatically and intelligently without human intervention, thereby realizing the intelligent collection of the completion status of each node of flight support. Compared with the traditional manual operation method, it can avoid the interference of human factors and reduce the delay error caused by manual entry, thereby improving the accuracy of the obtained support information. At the same time, it can also reduce the consumption of manpower and material resources in the operation of aviation support nodes, thereby improving the efficiency and accuracy of flight support, reducing the consumption of manpower and material resources, and helping the airport to dispatch flight support and handle abnormal situations.

[0125] In some embodiments, in an applicable scenario provided by the present invention, combined with Figure 1 and Figure 4 As shown, after executing step S14, the intelligent monitoring method of the flight support node provided by the present invention may further include the following steps:

[0126] S18. Save the operation process status and operation time of the first flight support node into a relational database in a structured manner.

[0127] In some embodiments, the code of any aircraft stand can be obtained, and then structured data is generated based on the operation process status, operation time and code of the first flight support node. The structured data uses the code of any aircraft stand as the primary key, and saves the operation process status and operation time of the first flight support node at the same time, and finally saves the structured data into a relational database.

[0128] For example, the operation process status, operation time and code of the first flight support node may be One-Hot encoded to generate structured data, such as generating structured data similar to 000111001. The relational database may be a MySQL relational database.

[0129] In the present invention, by adopting a structured method to save the operation process status and operation time of the first flight support node currently executed by any aircraft stand into a relational database, data support can be provided for the intelligent management and efficient operation of the airport.

[0130] In some embodiments, in an applicable scenario provided by the present invention, combined with Figure 1 and Figure 4 As shown, the intelligent monitoring method of the flight support node provided by the present invention may also include the following steps:

[0131] S19. Play the multi-view video stream through the video platform, and display the operation process status and operation time of the first flight support node.

[0132] In specific implementation, the video platform can obtain the multi-view video stream of any camera position through the streaming media server, parse the real-time video stream and play it. The video platform can also use the front-end and back-end interfaces to access the database, obtain the structured data corresponding to the first flight support node, and display the operation process status and operation time of the first flight support node based on the structured data.

[0133] As an example, the display interface of the video platform can also be used to add, modify, and mask aircraft stands, and query the support progress of a certain aircraft stand or flight.

[0134] In the present invention, a multi-view video stream is played through a video platform, and the operation process status and operation time of the first flight support node are displayed, so that the control personnel of the airport control center can observe the detailed information of the flight of each parking position in real time, which is helpful for the airport to dispatch the flight support and handle abnormal situations, such as overtime operation warning.

[0135] Based on the same inventive concept, the present invention also provides an intelligent monitoring device for flight support nodes, such as Figure 5 As shown, the intelligent monitoring device 20 of the flight support node may include:

[0136] The target detection unit 21 is used to obtain a multi-view video stream corresponding to any aircraft position in the airport apron; the target detection model based on the yolov5 network is used to intelligently identify the video frames of the multi-view video stream at different times, and determine the status information of the target aircraft and the target object located in any aircraft position; the target object includes one or more of various types of support vehicles, aircraft wheels, cargo doors and cabin doors; based on the status information of the target aircraft and the target object and the flight support node operation process, determine the first flight support node currently executed at any aircraft position, and update the flight support node of any aircraft position to the first flight support node; the flight support node operation process includes the time sequence relationship between each flight support node;

[0137] The monitoring unit 22 is used to determine the detection category to which the first flight support node belongs, and based on the detection category, monitor the operation process status of the first flight support node in a preset manner, and record the operation time of the first flight support node; if it is determined that the first flight support node is the last flight support node in the flight support node operation process, it is determined that all flight support nodes corresponding to any aircraft stand have been monitored; or, if it is determined that the first flight support node is not the last flight support node in the flight support node operation process, return to obtain the multi-view video stream corresponding to any aircraft stand in the airport apron through the target detection unit 21, and continue to monitor the flight support node of any aircraft stand.

[0138] In a possible design, the target detection unit 21 is specifically used for:

[0139] The multi-view video stream transmitted by the streaming media server is obtained; the multi-view video stream is collected by several video acquisition terminals monitoring different viewpoints of any camera position and saved in real time to the streaming media server.

[0140] In a possible design, the target detection model based on the yolov5 network adopts the yolov5 network as the basic structure, and uses the nearest neighbor association algorithm and the adjacent frame difference method to judge the motion state of the target aircraft and the target object; the target detection unit 21 is specifically used for:

[0141] The yolov5 network based on the target detection model of the yolov5 network is used to identify the video frames of the multi-view video stream at different times, and the positions of the target aircraft and the target object in the video frames at different times are output;

[0142] Based on the positions of the target aircraft and the target object in the video frames at different times, the nearest neighbor association algorithm and adjacent frame difference method of the target detection model based on the yolov5 network are used to determine the position changes of the target aircraft and the target object between the video frames at different times;

[0143] Based on the position changes of the target aircraft and the target object between the video frames at different times, the state information of the target aircraft and the target object located in any camera position is determined.

[0144] In a possible design, the flight support node operation process also includes a detection category of each flight support node, and the detection category is one of a target detection category, a pedestrian recognition and direction judgment category, and a motion detection category; the monitoring unit 22 includes an optical flow calculation module 221, a pedestrian attribute analysis module 222, and a motion detection module 223; the monitoring unit 22 is specifically used for:

[0145] Based on the flight support node operation process, determine the detection category of the first flight support node;

[0146] If the detection category of the first flight support node is the target detection category, the docking relationship between the target aircraft and the target object is monitored based on the status information of the target aircraft and the target object from the target detection unit 21; the operation process status of the first flight support node is monitored based on the docking relationship;

[0147] Alternatively, if the detection category of the first flight support node is pedestrian recognition and direction judgment, the target object includes the passenger elevator car, and the target detection unit 21 uses the yolov5 network of the target detection model based on the yolov5 network to adaptively generate the optical flow calculation area and the pedestrian detection area in the video frames of the multi-view video stream at different times; the optical flow calculation module 221 is used to perform optical flow calculation on the optical flow calculation area to determine the up and down status of the pedestrians on the passenger elevator car, and the pedestrian attribute analysis module 222 is used to perform pedestrian identity recognition on the pedestrian detection area to determine the identity attributes of the pedestrians on the passenger elevator car; based on the up and down status of the pedestrians and the pedestrian identity attributes, the operation process status of the first flight support node is monitored;

[0148] Alternatively, if the detection category of the first flight support node is action detection, the yolov5 network of the target detection model based on the yolov5 network of the target detection unit 21 is used to adaptively generate action detection areas in the video frames of the multi-view video stream at different times; the action detection module 223 is used to perform personnel support operation action detection on the action detection area to determine the current support operation action of the personnel in the first flight support node; based on the current support operation action, the operation process status of the first flight support node is monitored.

[0149] In a possible design, the optical flow calculation module 221 is specifically used to: generate a corner point distribution area of ​​the optical flow calculation area based on the Shi-Tomasi algorithm; the corner point distribution area is a rectangular area formed by the 1 / 5 positions on both sides of the upper left corner and the 1 / 5 positions on both sides of the lower right corner of the passenger elevator; use the optical flow algorithm to perform optical flow calculation on the corner point distribution area to obtain a sparse optical flow in the optical flow calculation area; if the vertical movement of the optical flow in the optical flow calculation area is determined to be upward based on the sparse optical flow, then the up and down state of the pedestrian on the passenger elevator is determined to be upward, or if the vertical movement of the optical flow in the optical flow calculation area is determined to be downward based on the sparse optical flow, then the up and down state of the pedestrian on the passenger elevator is determined to be downward;

[0150] The pedestrian attribute analysis module 222 is specifically used for: performing pedestrian recognition in the pedestrian detection area based on the target detection model of the EfficientDet algorithm, extracting the unified feature vector of pedestrians in the pedestrian detection area using the PCB ReID network, and calculating the maximum cosine similarity between the unified feature vectors of pedestrians; and determining the pedestrian identity attributes of the people on the passenger elevator based on the maximum cosine similarity.

[0151] In a possible design, the action detection module 223 is specifically used for:

[0152] A dual-stream based action recognition network is used to detect personnel support operation actions in the action detection area, and the confidence of the support operation actions occurring in the video frames of the multi-view video stream at different times is output; the support operation action corresponding to the maximum confidence is determined as the current support operation action of the personnel in the first flight support node.

[0153] In a possible design, the monitoring unit 22 is further used for:

[0154] Get the code of any camera position;

[0155] Generate structured data based on the operation process status, operation time and code of the first flight support node; the structured data uses the number of any aircraft position as the primary key, and saves the operation process status and operation time of the first flight support node at the same time;

[0156] Store structured data in a relational database.

[0157] In a possible design, the monitoring unit 22 is further used for:

[0158] The multi-view video stream is played through the video platform, and the operation process status and operation time of the first flight support node are displayed.

[0159] The intelligent monitoring device 20 of the flight support node in the embodiment of the present invention is similar to the above Figure 1 and Figure 4The intelligent monitoring method of the flight support node shown is an invention based on the same concept. Through the above detailed description of the intelligent monitoring method of the flight support node, those skilled in the art can clearly understand the implementation process of the intelligent monitoring device 20 of the flight support node in this embodiment, so for the sake of brevity of the specification, it will not be repeated here.

[0160] Based on the same inventive concept, the present invention also provides an intelligent monitoring device for flight support nodes, such as Figure 6 As shown, the intelligent monitoring device 30 of the flight support node may include: at least one memory 31 and at least one processor 32. Among them:

[0161] At least one memory 31 is used to store one or more programs.

[0162] When one or more programs are executed by at least one processor 32, the above Figure 1 and Figure 4 The intelligent monitoring method of the flight support node is shown.

[0163] The intelligent monitoring device 30 of the flight support node may also optionally include a communication interface, and the communication interface is used for communicating with external devices and exchanging data.

[0164] It should be noted that the memory 31 may include a high-speed RAM memory, and may also include a nonvolatile memory (nonvolatile memory), such as at least one disk memory.

[0165] In a specific implementation process, if the memory 31, the processor 32 and the communication interface are integrated on a chip, the memory 31, the processor 32 and the communication interface can communicate with each other through an internal interface. If the memory 31, the processor 32 and the communication interface are implemented independently, the memory 31, the processor 32 and the communication interface can be connected to each other through a bus and communicate with each other.

[0166] Based on the same inventive concept, the present invention also provides an intelligent monitoring system for flight support nodes, such as Figure 7 As shown, the intelligent monitoring system 40 of the flight support node may include:

[0167] Several video acquisition terminals 41, streaming media servers 42, intelligent monitoring devices 43 of flight support nodes, relational databases 44 and video platforms 45 are located at different viewing angles of any camera position. The streaming media servers 42 are respectively connected to the video acquisition terminals 41 and the intelligent monitoring devices 43 of flight support nodes, and the relational databases 44 are respectively connected to the intelligent monitoring devices 43 of flight support nodes and the video platforms 45.

[0168] The video acquisition terminal 41 is used to acquire the video stream of the corresponding viewing angle of any camera position, and transmit the acquired video stream to the streaming media server 42 in real time;

[0169] Streaming media server 42, used to store multi-view video streams of any camera position;

[0170] The intelligent monitoring device 43 of the flight support node is used to achieve the above Figure 1 and Figure 4 The intelligent monitoring method of the flight support node shown;

[0171] The relational database 44 is used to obtain the operation process status and operation time of the first flight support node from the intelligent monitoring device 43 of the flight support node, and obtain the code of any aircraft position, generate structured data based on the operation process status, operation time and code of the first flight support node, and save the structured data;

[0172] The video platform 45 is used to play multi-view video streams and display the operation process status and operation time of the first flight support node.

[0173] Through the above detailed description of the intelligent monitoring method of the flight support node, those skilled in the art can clearly understand the implementation process of the intelligent monitoring system 40 of the flight support node in this embodiment, so for the sake of brevity of the specification, it will not be repeated here.

[0174] Based on the same inventive concept, the present invention also provides a computer-readable storage medium, which can store at least one program. When the at least one program is executed by a processor, the above Figure 1 and Figure 4 The intelligent monitoring method of the flight support node is shown.

[0175] It should be understood that a computer-readable storage medium is any data storage device that can store data or programs, which can then be read by a computer system. Examples of computer-readable storage media include read-only memory, random access memory, CD-ROM, HDD, DVD, magnetic tape, and optical data storage devices, etc.

[0176] The computer readable storage medium can also be distributed over network coupled computer systems so that the computer readable code is stored and executed in a distributed fashion.

[0177] The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, radio frequency (RF), etc., or any suitable combination of the above.

[0178] The above-mentioned embodiments only express several implementation methods of the present invention, and the description is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. An intelligent monitoring method for flight support nodes, characterized in that: include: Get the multi-view video stream corresponding to any aircraft position in the airport apron; Using a target detection model based on the yolov5 network to intelligently identify video frames at different times of the multi-view video stream, determine the state information of the target aircraft and the state information of the target object located in any of the aircraft positions; the target object includes one or more of various types of support vehicles, aircraft wheels, cargo doors and cabin doors; Based on the status information of the target aircraft and the status information of the target object and the flight support node operation process, determine the first flight support node currently executed by any one of the aircraft stands, and update the flight support node of any one of the aircraft stands to the first flight support node; the flight support node operation process includes the time sequence relationship between each flight support node; Determine the detection category to which the first flight support node belongs, and based on the detection category, monitor the operation process status of the first flight support node in a preset manner, and record the operation time of the first flight support node; If it is determined that the first flight support node is the last flight support node in the flight support node operation process, it is determined that all flight support nodes corresponding to any one of the aircraft stands have been monitored; or, If it is determined that the first flight support node is not the last flight support node in the flight support node operation process, return to execute obtaining a multi-view video stream corresponding to any aircraft stand in the airport apron, and continue to monitor the flight support node of any aircraft stand; The flight support node operation process further includes the detection category of each flight support node, where the detection category is one of a target detection category, a pedestrian recognition and direction judgment category, and a motion detection category; determining the detection category of the first flight support node, and based on the detection category, monitoring the operation process status of the first flight support node in a preset manner, including: Determining a detection category of the first flight support node based on the flight support node operation process; If the detection category of the first flight support node is the target detection category, the docking relationship between the target aircraft and the target object is monitored based on the state information of the target aircraft and the state information of the target object; the operation process state of the first flight support node is monitored based on the docking relationship; Alternatively, if the detection category of the first flight support node is the pedestrian recognition and direction judgment category, the target object includes a passenger elevator car, and the yolov5 network of the target detection model based on the yolov5 network adaptively generates optical flow calculation areas and pedestrian detection areas in the video frames of the multi-view video stream at different times; the corner point distribution area of ​​the optical flow calculation area is generated based on the Shi-Tomasi algorithm, and the corner point distribution area is a rectangular area formed by the 1 / 5 position on both sides of the upper left corner and the 1 / 5 position on both sides of the lower right corner of the passenger elevator car; the optical flow algorithm is used to perform optical flow calculation on the corner point distribution area to obtain the sparse optical flow of the optical flow calculation area; if the vertical movement of the optical flow in the optical flow calculation area is determined to be upward based on the sparse optical flow, then the up and down state of the pedestrian on the passenger elevator car is determined to be upward, or, if the vertical movement of the optical flow in the optical flow calculation area is determined to be downward based on the sparse optical flow, then the up and down state of the pedestrian on the passenger elevator car is determined to be downward; and, the pedestrian identity is recognized in the pedestrian detection area based on the target detection model of the efficientdet algorithm, and PCB is used The ReID network extracts the unified feature vector of pedestrians in the pedestrian detection area, and calculates the maximum cosine similarity between the unified feature vectors of pedestrians; based on the maximum cosine similarity, determines the pedestrian identity attribute of the person on the passenger elevator; based on the pedestrian's up and down state and the pedestrian identity attribute, monitors the operation process status of the first flight support node; Alternatively, if the detection category of the first flight support node is the action detection category, the yolov5 network based on the target detection model of the yolov5 network adaptively generates action detection areas in the video frames of the multi-view video stream at different times; personnel support operation action detection is performed on the action detection area to determine the current support operation action of the personnel in the first flight support node; based on the current support operation action, the operation process status of the first flight support node is monitored.

2. The method according to claim 1, characterized in that Get the multi-view video stream corresponding to any aircraft position in the airport apron, including: The multi-view video stream transmitted by the streaming media server is obtained; the multi-view video stream is collected by a plurality of video acquisition terminals monitoring different viewing angles of any camera position and is saved in real time to the streaming media server.

3. The method according to claim 1, characterized in that The target detection model based on the yolov5 network adopts the yolov5 network as the basic structure, uses the nearest neighbor association algorithm and the adjacent frame difference method to judge the motion state of the target aircraft and the target object; uses the target detection model based on the yolov5 network to intelligently identify the video frames at different times of the multi-view video stream, and determines the state information of the target aircraft and the state information of the target object located in any of the camera positions, including: The yolov5 network of the target detection model based on the yolov5 network is used to identify the video frames of the multi-view video stream at different times, and the positions of the target aircraft and the target object in the video frames at different times are output; Based on the positions of the target aircraft and the target object in the video frames at different times, the nearest neighbor association algorithm and adjacent frame difference method of the target detection model based on the yolov5 network are used to determine the position change of the target aircraft and the target object between the video frames at different times; Based on the position changes of the target aircraft and the target object between the video frames at different times, the state information of the target aircraft and the state information of the target object located in any one of the aircraft positions are determined.

4. The method according to claim 1, characterized in that Performing personnel support operation action detection on the action detection area to determine the current support operation action of the personnel in the first flight support node includes: A dual-stream based action recognition network is used to detect personnel support operation actions in the action detection area, and the confidence of the support operation actions occurring in the video frames of the multi-view video stream at different times is output; the support operation action corresponding to the maximum confidence is determined as the current support operation action of the personnel in the first flight support node.

5. The method according to any one of claims 1 to 4, characterized in that: After determining the detection category to which the first flight support node belongs, and based on the detection category, monitoring the operation process status of the first flight support node in a preset manner, and recording the operation time of the first flight support node, the method further includes: Obtaining a code of any of the camera positions; Generate structured data based on the operation process status and operation time of the first flight support node and the code; the structured data uses the number of any one of the aircraft positions as a primary key and saves the operation process status and operation time of the first flight support node; The structured data is stored in a relational database.

6. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: The multi-view video stream is played through a video platform, and the operation process status and operation time of the first flight support node are displayed.

7. An intelligent monitoring device for a flight support node, characterized in that: include: A target detection unit is used to obtain a multi-view video stream corresponding to any aircraft position in an airport apron; use a target detection model based on a yolov5 network to intelligently identify video frames at different times of the multi-view video stream, and determine the state information of a target aircraft and a target object located in any aircraft position; the target object includes one or more of various types of support vehicles, aircraft wheels, cargo doors, and cabin doors; based on the state information of the target aircraft and the state information of the target object and the flight support node operation process, determine the first flight support node currently executed by any aircraft position, and update the flight support node of any aircraft position to the first flight support node; The flight support node operation process includes the timing relationship between each flight support node; a monitoring unit, configured to determine a detection category to which the first flight support node belongs, and based on the detection category, monitor the operation process status of the first flight support node in a preset manner, and record the operation time of the first flight support node; If it is determined that the first flight support node is the last flight support node in the flight support node operation process, it is determined that all flight support nodes corresponding to any one of the aircraft positions have been monitored; or, if it is determined that the first flight support node is not the last flight support node in the flight support node operation process, return to obtain the multi-view video stream corresponding to any one of the aircraft positions in the airport apron through the target detection unit, and continue to monitor the flight support node of any one of the aircraft positions; The flight support node operation process also includes the detection category of each flight support node, and the detection category is one of target detection, pedestrian recognition and direction judgment, and motion detection; the monitoring unit includes an optical flow calculation module, a pedestrian attribute analysis module, and a motion detection module; the monitoring unit is specifically used for: Determining a detection category of the first flight support node based on the flight support node operation process; If the detection category of the first flight support node is the target detection category, based on the state information of the target aircraft and the state information of the target object from the target detection unit, monitor the docking relationship between the target aircraft and the target object; based on the docking relationship, monitor the operation process status of the first flight support node; Alternatively, if the detection category of the first flight support node is the pedestrian recognition and direction judgment category, the target object includes a passenger elevator, and the target detection unit uses a yolov5 network based on a yolov5 network target detection model to adaptively generate an optical flow calculation area and a pedestrian detection area in the video frames of the multi-view video stream at different times; the corner point distribution area of ​​the optical flow calculation area is generated based on the Shi-Tomasi algorithm, and the corner point distribution area is a rectangular area formed by the 1 / 5 position on both sides of the upper left corner and the 1 / 5 position on both sides of the lower right corner of the passenger elevator; the optical flow algorithm is used to perform optical flow calculation on the corner point distribution area to obtain a sparse optical flow of the optical flow calculation area; if the vertical movement of the optical flow in the optical flow calculation area is determined to be upward based on the sparse optical flow, then the up and down state of the pedestrian on the passenger elevator is determined to be upward, or, if the vertical movement of the optical flow in the optical flow calculation area is determined to be downward based on the sparse optical flow, then the up and down state of the pedestrian on the passenger elevator is determined to be downward; and, the pedestrian identification is performed on the pedestrian detection area based on the target detection model of the efficientdet algorithm, and PCB is used The ReID network extracts the unified feature vector of pedestrians in the pedestrian detection area, and calculates the maximum cosine similarity between the unified feature vectors of pedestrians; based on the maximum cosine similarity, determines the pedestrian identity attribute of the person on the passenger elevator; based on the pedestrian's up and down state and the pedestrian identity attribute, monitors the operation process status of the first flight support node; Alternatively, if the detection category of the first flight support node is the action detection category, the action detection area in the video frames of the multi-view video stream at different times is adaptively generated through the yolov5 network of the target detection model based on the yolov5 network of the target detection unit; the action detection module is used to perform personnel support operation action detection on the action detection area to determine the current support operation action of the personnel in the first flight support node; based on the current support operation action, the operation process status of the first flight support node is monitored.

8. An intelligent monitoring system for flight support nodes, characterized in that: include: Several video acquisition terminals, streaming media servers, intelligent monitoring devices of flight support nodes, relational databases and video platforms located at different viewing angles of any aircraft position, wherein the streaming media servers are respectively connected to the video acquisition terminals and the intelligent monitoring devices of the flight support nodes, and the relational databases are respectively connected to the intelligent monitoring devices of the flight support nodes and the video platform; wherein, The video acquisition terminal is used to acquire the video stream of the corresponding viewing angle of any camera position, and transmit the acquired video stream to the streaming media server in real time; The streaming media server is used to store the multi-viewing angle video stream of any camera position; The intelligent monitoring device of the flight support node is used to execute the method according to any one of claims 1 to 6; The relational database is used to obtain the operation process status and operation time of the first flight support node from the intelligent monitoring device of the flight support node, and obtain the code of any one of the aircraft positions, generate structured data based on the operation process status and operation time of the first flight support node and the code, and save the structured data; The video platform is used to play the multi-view video stream and display the operation process status and operation time of the first flight support node.

Images