A method for detecting the state of a covered bridge based on deep learning and feature matching

Abstract

This invention discloses a boarding bridge status detection method based on deep learning and feature matching, comprising: acquiring image data from on-site cameras at the airport; detecting the aircraft, boarding bridge, and boarding bridge docking status on the acquired image data; determining whether the aircraft has reached its parking position using the detection results of consecutively spaced frames; and determining the starting position of the boarding bridge after the aircraft has parked, using the detection information from the image data and the maximum distance between the aircraft and the boarding bridge. This invention can determine the boarding bridge status from multiple angles and has good scene generalization; it can also effectively determine the boarding bridge status even when the boarding bridge is obstructed, demonstrating good robustness; it can also effectively determine the boarding bridge status in low-visibility conditions such as foggy or rainy weather at the airport; this method does not require additional equipment or manual delineation of the judgment area, saving subsequent maintenance costs.

Description

Technical Field

[0001] This invention relates to the field of civil aviation apron safety management, specifically to a method for detecting the status of boarding bridges based on deep learning and feature matching. Background Technology

[0002] Aircraft jet bridges, also known as boarding bridges or apron jet bridges, are essential equipment for passenger boarding and disembarking, allowing passengers to board and disembark safely and comfortably regardless of weather or temperature. Real-time monitoring of jet bridges using existing airport surveillance videos can automatically detect the start and end times of jet bridge movements near and away from aircraft during flight operations. This allows airport operators to monitor critical processes in real time. Currently, jet bridge status is primarily determined by signals from onboard systems. However, the number of new jet bridges equipped with onboard systems is limited, and retrofitting them is costly. Furthermore, existing onboard systems may malfunction or return incorrect signals, leading to errors in operator judgment. Deep learning-based jet bridge status detection can effectively mitigate these risks.

[0003] In an apron scenario, after the aircraft is parked in its designated position, operators need to maneuver the jet bridge from the boarding gate to the aircraft cabin door to facilitate passenger access. To enable support personnel to obtain real-time status information about the jet bridge (direction of movement, docking status, and real-time images), we must consider not only potential obstructions (no more than 60%) caused by aircraft or other objects under various monitoring conditions, but also different weather conditions (rain, fog, etc.) to ensure the jet bridge's status can be detected under all circumstances. This allows operators to monitor critical flight support processes in real time and minimizes the occurrence of unnecessary accidents.

[0004] Chinese Patent (CN110210427A) provides a system and method for detecting the working status of a covered bridge based on image processing technology. The method has the following problems: (1) The method cannot cope with complex weather scenarios, such as fog or rain causing the covered bridge to be blurred, and the convolutional network will not be able to locate the position of the covered bridge; (2) The method cannot cope with situations where the covered bridge may be occluded in multiple scenes and at multiple angles, and the convolutional network will not be able to locate the position of the covered bridge; (3) When the scene environment changes, such as the camera being moved or interference from rain, the method cannot identify the feature area of ​​the covered bridge through frame difference; (4) The method finally uses the center line of the feature area of ​​the covered bridge to divide the area to determine the working status of the covered bridge. The area division of the method needs to be manually set in advance. Whenever the area changes or needs to be adjusted, the area needs to be manually divided. The subsequent maintenance requires a lot of manpower and time, which is clumsy and inflexible. Summary of the Invention

[0005] This invention aims to at least partially solve one of the technical problems in the aforementioned technologies. Therefore, the purpose of this invention is to provide a method for detecting the state of a covered walkway based on deep learning and feature matching. This method not only reduces the impact of external factors such as weather on the state of the covered walkway, but also effectively detects its state even when the walkway is partially obstructed, ensuring stable state detection that is less susceptible to external interference. Furthermore, it eliminates the need for subsequent human intervention and area division, significantly reducing maintenance costs.

[0006] The second objective of this invention is to propose a bridge state detection device based on deep learning and feature matching.

[0007] The third objective of this invention is to provide a computer device.

[0008] The fourth objective of this invention is to provide a computer-readable storage medium.

[0009] To achieve the above objectives, a first aspect of the present invention proposes a method for detecting the state of a covered bridge based on deep learning and feature matching, comprising:

[0010] Image data was obtained from on-site cameras at the airport;

[0011] The acquired image data is used to detect the aircraft, the boarding bridge, and the docking status of the boarding bridge.

[0012] The detection results of consecutive interval frames are used to determine whether the aircraft has reached the parking position;

[0013] The starting position of the jet bridge is determined based on the detection information from the image data and the maximum distance between the aircraft and the jet bridge after the aircraft comes to a stop.

[0014] The features corresponding to the determined starting position of the covered bridge are convolutionally encoded and saved;

[0015] The status of the jet bridge is determined by assessing the detected aircraft position and the starting position of the jet bridge.

[0016] Preferably, the step of detecting the aircraft, boarding bridge, and boarding bridge docking status from the acquired image data includes:

[0017] The video stream was acquired using the airport's camera equipment and then analyzed into individual images.

[0018] In this process, video streams acquired using airport cameras are analyzed using a YOLOv5 detection model. The detected objects are aircraft, jet bridges, and jet bridge docking.

[0019] Optionally, a detection box can be added to the object to be detected during the detection process.

[0020] Preferably, determining whether the aircraft has reached the parking position using the detection results of consecutively spaced frames includes:

[0021] Obtain aircraft inspection results based on docking status detection;

[0022] Determine whether the change in the center point of the detection box of the aircraft in consecutive interval frames exceeds a threshold. If it does not exceed the threshold, the aircraft has come to a stop; otherwise, the aircraft is in motion.

[0023] Preferably, determining the starting position of the jet bridge based on the detection information from the image data and the maximum distance between the aircraft and the jet bridge after the aircraft has come to a stop includes:

[0024] The target detection model YOLOv5 is used to detect target aircraft, boarding bridge and boarding bridge status in the parsed video stream image data, and to obtain the location information of the detected targets.

[0025] By using the detected aircraft information, the difference between consecutive interval frames is used to determine whether the aircraft is stationary.

[0026] After the aircraft comes to a stop, the Euclidean distances D1 and D2 between the detected aircraft and the boarding bridge are calculated in the preceding and following frames, respectively, using the detected aircraft and boarding bridge information. The absolute values ​​abs of D1 and D2 are also calculated.

[0027] The obtained abs are compared with the set threshold 1;

[0028] Extract and save the detection frame of the initial state of the bridge.

[0029] Preferably, the step of performing convolutional encoding and saving the features corresponding to the determined starting position of the covered bridge includes:

[0030] The detection box features of aircraft, boarding bridges, and boarding bridge docking status detection are encoded by convolutional neural networks and stored in the database. A boarding bridge status classification model is trained, and the features before the fully connected layer are extracted by the boarding bridge status classification model as the encoded features of the boarding bridge.

[0031] In the convolutional neural network, the encoding network uses the ResNet18 network, and the loss function is CircleLoss.

[0032] Preferably, the step of determining the boarding bridge's operational status based on the target aircraft's detected position and the boarding bridge's starting position, according to image data, includes:

[0033] The status of the covered bridge can be divided into: the initial stationary state, the approaching state, the docking state, and the evacuation state.

[0034] After determining that the covered bridge is initially stationary, the process begins to assess its state when it is moving and stationary.

[0035] To achieve the above objectives, a second aspect of the present invention proposes a bridge state detection device based on deep learning and feature matching, comprising:

[0036] The data acquisition module is used to acquire image data from the on-site camera;

[0037] The data acquisition module is also used to detect the aircraft, boarding bridge, and boarding bridge docking status in the acquired images;

[0038] The data analysis module is used to determine whether the aircraft has reached the parking position by using the detection results of consecutive interval frames;

[0039] The data analysis module is also used to determine the starting position of the jet bridge based on the detection information of the image data and the maximum distance between the aircraft and the jet bridge after the aircraft has come to a stop.

[0040] The data analysis module is also used to perform convolutional encoding on the features corresponding to the determined starting position of the covered bridge and save them;

[0041] The data analysis module is also used to determine the status of the boarding bridge based on the target aircraft position detected by the image data and the starting position of the boarding bridge, thus determining the working status of the boarding bridge.

[0042] To achieve the above objectives, a third aspect of the present invention provides a computer device, which is a physical device, comprising:

[0043] The processor and memory are communicatively connected to the processor.

[0044] The memory is used to store at least one executable instruction executed by the processor, which executes the executable instruction to implement the bridge state detection method based on deep learning and feature matching as described above.

[0045] To achieve the above objectives, a fourth aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the bridge state detection method based on deep learning and feature matching as described above.

[0046] Compared with the prior art, the beneficial effects of the present invention are:

[0047] 1. This solution can determine the status of the covered bridge from multiple perspectives and has good scenario generalization ability;

[0048] 2. It can effectively determine the status of the covered bridge even when it is obstructed (less than 60%), demonstrating good robustness;

[0049] 3. When visibility is low (less than 150m) in foggy or rainy weather at the airport, the jet bridge status detection can also make a good judgment.

[0050] 4. This method does not require additional equipment or manual delineation of the judgment area, thus saving subsequent maintenance costs. Attached Figure Description

[0051] Figure 1 The main flowchart of the algorithm for a bridge state detection method based on deep learning and feature matching provided in an embodiment of the present invention is shown below.

[0052] Figure 2 A flowchart illustrating the determination of the starting position of a covered bridge using a method for detecting the state of a covered bridge based on deep learning and feature matching, provided in an embodiment of the present invention.

[0053] Figure 3 A flowchart illustrating the determination of the starting position of a covered bridge using a method for detecting the state of a covered bridge based on deep learning and feature matching, provided in an embodiment of the present invention.

[0054] Figure 4 A flowchart illustrating the process of determining the movement state of a covered bridge using a method based on deep learning and feature matching, provided in an embodiment of the present invention.

[0055] Figure 5 A flowchart illustrating the static state determination of a covered bridge using a method for detecting the state of a covered bridge based on deep learning and feature matching, provided for an embodiment of the present invention.

[0056] Figure 6 Inspection diagrams of the boarding bridge and aircraft provided in this embodiment of the invention;

[0057] Figure 7 This is a docking status detection diagram provided for an embodiment of the present invention. Detailed Implementation

[0058] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0059] The method in this embodiment is executed by a terminal, which can be a mobile phone, tablet computer, PDA, laptop or desktop computer, etc. Of course, it can also be other devices with similar functions, and this embodiment does not limit them.

[0060] Please see Figures 1 to 7This invention provides an adaptive bridge operation status detection method based on deep learning and feature matching. The method is applied to determine the operation status of a bridge and includes:

[0061] Step S1: Acquire image data from the on-site camera.

[0062] Step S2: Detect the aircraft, the boarding bridge, and the docking status of the boarding bridge in the acquired image.

[0063] Furthermore, S1 and S2 also include:

[0064] The video stream was acquired using the airport's camera equipment and then analyzed into individual images.

[0065] like Figure 6 and Figure 7 As shown, Figure 6 The inspection diagrams for the jet bridge and aircraft were provided. Figure 7 A docking status detection map is provided. The video stream is obtained using the airport's camera equipment. To improve the detection effect and performance, the image is analyzed using the YOLOv5 detection model. The detection objects are divided into three categories: aircraft, boarding bridge, and boarding bridge docking detection. Detection boxes are added to the objects.

[0066] YOLOv5 is a single-stage target detection algorithm that boasts excellent performance and accuracy, and is widely used in industry. This solution utilizes this algorithm to achieve good results in both speed and accuracy. The main structure of YOLOv5 includes the input terminal, backbone, neck, and head, as detailed below:

[0067] I. Input: During the model training phase, some improvement ideas were proposed, mainly including Mosaic data augmentation, adaptive anchor box calculation, and adaptive image scaling;

[0068] II. Backbone: Integrates some new ideas from other detection algorithms. The previous Yolov5 version used the Focus structure and CSP structure. In version 6.0, Conv(k=6,s=2,p=2) convolution replaced Focus, and SPP was replaced with SPPF and placed at the end of the backbone, resulting in better performance.

[0069] 3. Neck: Object detection networks often insert some layers between the Backbone and the final Head output layer. Yolov5 added the FPN+PAN structure. This scheme adopts a structure with four different scale feature outputs. Specifically, the input is 640*640*3, and the dimensions of the generated multi-scale feature maps are: 160*160*256, 80*80*512, 40*40*768, and 20*20*1024.

[0070] IV. Head: The anchor box mechanism of the output layer is the same as that of YOLOv4. The main improvements are the loss function GIOU_Loss during training and the DIOU_nms for selecting predicted boxes.

[0071] Step S3: Use the detection results of consecutive interval frames to determine whether the aircraft has reached the parking position.

[0072] Furthermore, such as Figure 2 As shown, Figure 2 A flowchart for determining the starting position of a covered bridge using a method for detecting the state of a covered bridge based on deep learning and feature matching is provided. Step S3 includes at least the following:

[0073] S31. Collect image data and obtain aircraft detection results based on docking status detection;

[0074] S32. Determine whether the change in the center point of the detection box of the aircraft exceeds the threshold within at least 20 consecutive frames. If it does not exceed the threshold, the aircraft has come to a stop; otherwise, the aircraft is in motion.

[0075] Step S4: Determine the starting position of the jet bridge based on the detection information of the image data and the maximum distance between the aircraft and the jet bridge after the aircraft comes to a stop.

[0076] Furthermore, such as Figure 3 As shown, Figure 3 A flowchart for determining the starting position of a covered bridge using a method for detecting the state of a covered bridge based on deep learning and feature matching is provided. Step S4 includes at least the following:

[0077] S41. Use the target detection model Yolov5 to detect the target aircraft, the boarding bridge and the state of the boarding bridge in the parsed video stream image data, and obtain the location information of the detected targets.

[0078] S42. Using the detected aircraft information, determine whether the aircraft is stationary by using the difference between consecutive interval frames.

[0079] S43. After the aircraft comes to a stop, use the detected aircraft and boarding bridge information to calculate the Euclidean distances D1 and D2 between the aircraft and boarding bridge in the preceding and following frames, and calculate the absolute values ​​abs of D1 and D2.

[0080] S44. Compare the obtained abs with the set threshold 1, specifically including the following steps:

[0081] s441. If abs > threshold 1, it means the bridge was still in a moving state at the beginning; then compare the sizes of D1 and D2:

[0082] s442. If D1>D2, it means that the jet bridge is moving towards the aircraft, and the jet bridge is determined to be in a close approach state at this time.

[0083] s443. If D1 < D2, it indicates that the jet bridge is moving away from the non - aggregation area. At this time, it is determined that the jet bridge is in the evacuation state;

[0084] s444. If abs < threshold 1, it means that the jet bridge is still in a stationary state at the beginning, and then it is determined whether the model detects the docking state of the jet bridge;

[0085] s445. If the docking state of the jet bridge is detected, it means that the jet bridge is in the docking state;

[0086] s446. If the docking state of the jet bridge is not detected, it means that the jet bridge is in a stationary state without working;

[0087] S45. Extract the position box of the detected starting state of the jet bridge and save it.

[0088] Step S5. Convolutionally encode the features corresponding to the determined starting position of the jet bridge and save them.

[0089] Furthermore, the step S5 at least further includes:

[0090] Encode the position box features of the aircraft, jet bridge, and jet bridge docking state detection through a convolutional neural network and save them to the database. To solve the influence brought by weather changes and partial occlusion of the jet bridge, a jet bridge state classification model is trained, and the features before the fully - connected layer are extracted as the encoded features of the jet bridge to reduce the influence brought by external changes.

[0091] Among them, in the convolutional neural network, the encoding network uses the Resnet18 network. To better train the feature expression ability of the network, the loss function uses Circle Loss during training.

[0092] Specifically, there are two basic paradigms for deep feature learning, namely learning using class labels and learning using positive - negative sample pairs. When using class labels, generally, a classification loss function (such as softmax + cross entropy) is used to optimize the similarity between samples and weight vectors; when using sample pairs, a metric loss function (such as triplet loss) is usually used to optimize the similarity between samples. Both functions maximize the intra - class similarity (s p ) and minimize the inter - class similarity (S n ); loss functions (such as triplet loss, softmax loss, and their variants) have similar optimization patterns. They will combine S n and S p into a similarity pair to optimize and try to reduce (S n - Sp ), in (S n -S p In ), increase S p Equivalent to reducing S n This symmetric optimization method is prone to the following two problems: lack of optimization flexibility and unclear convergence state.

[0093] A new algorithm based on Circle Loss is proposed to address the issues of inflexibility and ambiguous convergence states in the optimization of the two functions mentioned above. The formula is as follows:

[0094]

[0095] Where α j n and α j p S is a non-negative weighting factor, γ is a scaling factor; define S p The optimal value is O p S n The optimal value is O n And O n <O p When a similarity score deviates significantly from the optimal value, Circle Loss will assign it a larger weight for further adaptive optimization and updates. To allow each similarity score to adjust the learning magnitude according to the current optimization state, the self-pacing is defined as follows:

[0096]

[0097] in[·] + To ensure α is truncated to zero j n and α j p The non-negative Circle loss performs an independent weighting based on the value of each similarity category, allowing for different learning magnitudes and providing greater flexibility.

[0098] Step S6: Determine the status of the boarding bridge based on the detected aircraft position and the starting position of the boarding bridge to determine its working status.

[0099] Furthermore, step S6 includes at least:

[0100] S61. The state of the covered bridge can be divided into: the initial static state of the covered bridge, the approaching state of the covered bridge, the docking state of the covered bridge, and the evacuation state of the covered bridge. After the initial state of the covered bridge is determined, the state judgment of the covered bridge when it is moving and stationary begins.

[0101] S62. Determining the movement status of the covered bridge, such as... Figure 4 As shown, Figure 4 A flowchart illustrating the process of determining the movement state of a covered bridge based on deep learning and feature matching is provided. The specific determination steps are as follows:

[0102] s621. First, calculate the Euclidean distance D3 between the center points of the detected boarding bridge position information and aircraft position information;

[0103] s622. Based on the bridge detection information, calculate the similarity between the features of the current bridge detection box and the starting box;

[0104] s623. Compare whether the change in D3 between the previous and next frames exceeds the threshold T4 and whether the similarity S2 is less than the specified threshold T3;

[0105] s624. If the distance does not change and the features of the bridge detection box are similar to the features of the starting box, then the bridge is in a stationary state. Then, state docking and starting judgment are performed on it. Otherwise, the bridge is in a moving state. Then, approaching and leaving judgment are performed on it.

[0106] s625. Movement status judgment: If the distance increases, it is judged as a withdrawal state; otherwise, it is a closing state.

[0107] S63. Determining the stationary state of the covered bridge, such as Figure 5 As shown, Figure 5 A flowchart for determining the static state of a covered bridge using a method based on deep learning and feature matching is provided. The specific determination steps are as follows:

[0108] s631. After determining that the covered bridge is stationary, begin to determine the state of the covered bridge when it is stationary.

[0109] s632. First, encode the features of the current frame based on the starting position frame using F1;

[0110] s633, Calculate the similarity between the encoded features and the base database features (S1);

[0111] s634. Based on the detection results, calculate the distances BAD (and distance) and BBD (and begin distance) between the boarding bridge, the aircraft, and the starting position frame at this moment.

[0112] s635. If the similarity S1 is greater than the specified threshold T2 and the distance between the boarding bridge and the aircraft is greater than the distance between the boarding bridge and the starting position, it means that the features of the bounding box and the base library are very similar at this moment, and the boarding bridge is at the starting position. Otherwise, the boarding bridge is in a docking state.

[0113] s636. Update the current bridge frame to the starting position frame A and save it to the bottom library. Update and replace it with feature F.

[0114] In this embodiment, the method can determine the status of the boarding bridge from multiple angles and has good scene generalization; it can also effectively determine the status of the boarding bridge when it is obscured (less than 60%), showing good robustness; it can also determine the status of the boarding bridge well when the visibility is low (less than 150m) in foggy or rainy weather at the airport; the method does not require additional equipment or manual delineation of the judgment area, saving subsequent maintenance costs.

[0115] Based on the above embodiments, the present invention also provides a bridge state detection device based on deep learning and feature matching, for supporting the bridge state detection method based on deep learning and feature matching in the above embodiments. The bridge state detection device based on deep learning and feature matching includes:

[0116] The data acquisition module is used to acquire image data from the on-site camera;

[0117] The data acquisition module is also used to detect the aircraft, boarding bridge, and boarding bridge docking status in the acquired images;

[0118] The data analysis module is used to determine whether the aircraft has reached the parking position by using the detection results of consecutive interval frames;

[0119] The data analysis module is also used to determine the starting position of the jet bridge based on the detection information of the image data and the maximum distance between the aircraft and the jet bridge after the aircraft has come to a stop.

[0120] The data analysis module is also used to perform convolutional encoding on the features corresponding to the determined starting position of the covered bridge and save them;

[0121] The data analysis module is also used to determine the status of the boarding bridge based on the target aircraft position detected by the image data and the starting position of the boarding bridge, thus determining the working status of the boarding bridge.

[0122] Furthermore, the adaptive bridge working status detection device based on deep learning and feature matching can run the bridge status detection method based on deep learning and feature matching described above. For specific implementation, please refer to the method embodiment, which will not be repeated here.

[0123] Based on the above embodiments, the present invention also provides a computer device, the computer device comprising:

[0124] The processor and the memory are communicatively connected;

[0125] In this embodiment, the memory can be implemented in any suitable manner, for example, the memory can be a read-only memory, a hard disk drive, a solid-state drive, or a USB flash drive, etc.; the memory is used to store at least one executable instruction executed by the processor;

[0126] In this embodiment, the processor can be implemented in any suitable manner. For example, the processor can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) that can be executed by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers, etc.; the processor is used to execute the executable instructions to implement the bridge state detection method based on deep learning and feature matching as described above.

[0127] Based on the above embodiments, the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the bridge state detection method based on deep learning and feature matching as described above.

[0128] Those skilled in the art will recognize that the modules and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0129] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, equipment, and modules described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0130] In the several embodiments provided in this application, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or units may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices, or equipment, and may be electrical, mechanical, or other forms.

[0131] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0132] In addition, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0133] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program instructions, such as USB flash drives, portable hard drives, read-only storage servers, random access storage servers, magnetic disks, or optical disks.

[0134] Furthermore, it should be noted that the combination of the various technical features in this case is not limited to the combination methods described in the claims of this case or the combination methods described in the specific embodiments. All technical features described in this case can be freely combined or combined in any way, unless they contradict each other.

[0135] It should be noted that the above examples are merely specific embodiments of the present invention, and the present invention is obviously not limited to the above embodiments, with many similar variations. All modifications that can be directly derived or conceived by those skilled in the art from the content disclosed in this invention should fall within the protection scope of this invention.

[0136] The above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A gallery bridge state detection method based on deep learning and feature matching, characterized in that, The method comprises the following steps: obtaining image data from a live camera at an airport; detecting a plane, a jetty and a jetty docking state from the obtained image data; determining whether the plane has arrived at a parking position by using the detection results of consecutive interval frames; determining a starting position of the jetty according to the detection information of the image data and the maximum distance between the plane and the jetty after the plane is stationary, comprising: detecting a target plane, a jetty and a jetty state from the analyzed video stream image data by using a target detection model Yolov5 to obtain position information of the detected target; determining whether the plane is stationary by using the detection plane information and using a consecutive interval frame difference; after the plane is stationary, calculating the Euclidean distances D1 and D2 of the detected plane and jetty in the front and rear frames respectively, and calculating the absolute values abs of D1 and D2; comparing the obtained abs with a set threshold value 1; extracting and saving the detection frame of the detected starting state of the jetty; convolutionally encoding and saving the features corresponding to the determined starting position of the jetty, comprising: encoding and saving the features of the detection frame of the plane, the jetty and the jetty docking state detection to a database by using a convolutional neural network, and training a jetty state classification model, and extracting the features before the full connection layer as the encoding features of the jetty by using the jetty state classification model; wherein, in the convolutional neural network, the encoding network uses a Resnet18 network, and the loss function uses a Circle Loss. judging the jetty state according to the detected plane position and the starting position of the jetty to determine the working state of the jetty. 2.The deep learning and feature matching based gallery bridge state detection method according to claim 1, characterized in that, The method for detecting a plane, a jetty and a jetty docking state from the obtained image data comprises: obtaining a video stream by using a camera device at the airport, and analyzing the video stream into each image; wherein, in the step of obtaining a video stream by using a camera device at the airport, the image is detected by using a Yolov5 detection model, and the detection objects are a plane, a jetty and a jetty docking. 3.The deep learning and feature matching based gallery bridge state detection method of claim 1, wherein, The method for determining whether the plane has arrived at a parking position by using the detection results of consecutive interval frames comprises: obtaining a plane detection result according to the docking state detection; judging whether the center point of the detection frame of the plane in consecutive interval frames changes by more than a threshold value, if not, the plane has been parked stably, otherwise the plane is in a moving state. 4.The method of claim 1, wherein, The method for judging the jetty state according to the detected target plane position and the starting position of the jetty to determine the working state of the jetty comprises: the jetty state can be divided into: a jetty starting stationary state, a jetty approaching state, a jetty docking state and a jetty withdrawal state; after it is determined that the jetty is in a starting stationary state, the state of the jetty in a moving and stationary state is judged.

5. A gallery bridge state detection device based on deep learning and feature matching, characterized by, The method comprises the following steps: a data acquisition module is used to obtain image data from a live camera; the data acquisition module is also used to detect a plane, a jetty and a jetty docking state from the obtained image data; a data analysis module is used to determine whether the plane has arrived at a parking position by using the detection results of consecutive interval frames; The data analysis module is further configured to determine the starting position of the bridge based on the detection information of the image data and the maximum distance between the airplane and the bridge after the airplane is stationary, including: performing target airplane, bridge, and bridge state detection on the parsed video stream image data by a target detection model Yolov5 to obtain position information of the detected target; determining whether the airplane is stationary by using a continuous interval frame difference value based on the detected airplane information; after the airplane is stationary, calculating the Euclidean distances D1 and D2 of the airplane and the bridge in the previous and subsequent frames based on the detected airplane and bridge information, and calculating the absolute values abs of D1 and D2; comparing the obtained abs with a set threshold value 1; extracting and saving the detection frame of the detected starting state of the bridge; The data analysis module is further configured to perform convolutional coding on the features corresponding to the determined starting position of the bridge and save the features, including: encoding and saving the features of the detection frame of the airplane, bridge, and bridge docking state detection to a database by a convolutional neural network, and training a bridge state classification model, and extracting the features before the full connection layer of the bridge state classification model as the encoding features of the bridge; in the convolutional neural network, the encoding network uses a Resnet18 network, and the loss function uses a Circle Loss; The data analysis module is further configured to determine the working state of the bridge based on the position of the target airplane detected from the image data and the starting position of the bridge.

6. A computer device, comprising: The computer device comprises: a processor and a memory, which are in communication connection with the processor; the memory is configured to store executable instructions executed by the processor, and the processor is configured to execute the executable instructions to implement the bridge state detection method based on deep learning and feature matching according to any one of claims 1 to 4.

7. A computer readable storage medium characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the bridge state detection method based on deep learning and feature matching according to any one of claims 1 to 4.

Images