Visual tracking method and device for boarding bridge and related equipment

Abstract

The invention provides a visual tracking method and device for a boarding bridge and related equipment, and relates to the technical field of automatic control. The method comprises the following steps: in a docking control process of a boarding bridge and a cabin door, continuously acquiring multiple frames of cabin door images through a visual system carried on the boarding bridge; according to the motion information of the cabin door feature points in the target frame cabin door image, the size information of the tracking window in the target frame cabin door image is dynamically adjusted, and the target frame cabin door image is any frame cabin door image in the multiple frames of cabin door images; based on a visual tracking algorithm, cabin door feature points are extracted from the tracking window of the target frame cabin door image, and coordinate information is output. The problem that the feature points in the tracking window are lost due to the movement of the boarding bridge can be avoided, and the visual tracking effect is improved.

Description

Technical Field

[0001] This disclosure relates to the field of automation control technology, and in particular to a visual tracking method, apparatus and related equipment for boarding bridges. Background Technology

[0002] With the increasing automation in the aviation industry, the precise and efficient docking of boarding bridges and aircraft doors has become a key link in ensuring airport operational efficiency and safety. Visual guidance systems, due to their advantages such as cost control and strong environmental adaptability, have gradually replaced traditional manual observation and guidance methods, becoming the main control means for the automatic docking of boarding bridges with aircraft doors. However, their efficiency and accuracy are highly dependent on visual tracking algorithms.

[0003] Current mainstream boarding bridge visual guidance systems and supporting visual tracking technologies are difficult to fully adapt to the complex scenario requirements of boarding bridges connecting to aircraft doors, which can easily lead to the loss of some feature points within the tracking window and affect the accuracy of visual tracking results.

[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] This disclosure provides a visual tracking method, apparatus, and related equipment for boarding bridges, which at least to some extent overcomes the technical problem in related technologies where visual tracking algorithms using fixed-size tracking windows lose some feature points within the tracking window due to boarding bridge movement.

[0006] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part from practice of this disclosure.

[0007] According to one aspect of this disclosure, a visual tracking method for a boarding bridge is provided, comprising: acquiring multiple frames of cabin door images continuously collected by a vision system mounted on the boarding bridge; dynamically adjusting the size information of a tracking window in a target frame cabin door image based on motion information of cabin door feature points in the target frame cabin door image, wherein the target frame cabin door image is any one of the multiple frames of cabin door images; extracting coordinate information of cabin door feature points from the tracking window of the target frame cabin door image based on a visual tracking algorithm; determining the relative pose information of the cabin door and the boarding bridge based on the coordinate information of the cabin door feature points extracted from the target frame cabin door image; and controlling the boarding bridge to dock with the cabin door based on the relative pose information.

[0008] In some embodiments, dynamically adjusting the size information of the tracking window in the target frame hatch image based on the motion information of the hatch feature points in the target frame hatch image includes: determining the size information of the tracking window using the following formula. Size information of the tracking window in the frame hatch image: ; in, Indicates the first The hatch feature points in the frame image are Pixel coordinates along the axis; Indicates the first The hatch feature points in the frame image are Pixel coordinates along the axis; Indicates the first The width of the tracking window in the frame hatch image; Indicates the first The height of the tracking window in the frame hatch image; and These are two parameters that change dynamically with time / frame and are related to the motion information of the hatch feature points.

[0009] In some embodiments, the motion information of the hatch feature points includes at least: the velocity of the hatch feature points; wherein... The calculation formula is as follows: ; ; in, Indicates the baseline width of the tracking window; Indicates the vibration sensitivity coefficient; A velocity vector representing the characteristics of the hatch; Represents a function related to the velocity of a characteristic point of the hatch. This indicates that the vision system is acquiring the first... The velocity vector of the hatch feature points in a frame of hatch image; This indicates that the vision system is acquiring the first... The velocity vector of the hatch feature points in a frame of hatch image; Indicates the first Frame and the The duration of the frame interval between frames; express and The magnitude of the difference vector is the change in velocity.

[0010] In some embodiments, The calculation formula is as follows: ; or, ; in, Indicates whether the aspect ratio is fixed or variable; This indicates the reference height of the tracking window.

[0011] In some embodiments, the relative pose information includes: angular deviation and / or positional deviation; wherein, controlling the boarding bridge to dock with the cabin door according to the relative pose information includes: controlling the boarding bridge to perform rotation and / or forward movement according to the angular deviation and / or positional deviation.

[0012] In some embodiments, before extracting the coordinate information of hatch feature points from the tracking window of the target frame hatch image based on a visual tracking algorithm, the method further includes: determining multiple confidence evaluation factors for quantitatively evaluating the visual tracking results; fusing the multiple confidence evaluation factors to obtain a comprehensive confidence score; and determining whether to optimize the visual tracking results based on the value range of the comprehensive confidence score.

[0013] In some embodiments, determining whether to optimize the visual tracking result based on the comprehensive confidence score includes: if the comprehensive confidence score is less than a first threshold, then using a Kalman filter algorithm to predict the coordinate information of the hatch feature points in the target frame hatch image; if the comprehensive confidence score is greater than a second threshold, then directly extracting the coordinate information of the hatch feature points from the tracking window of the target frame hatch image based on the visual tracking algorithm, wherein the second threshold is greater than the first threshold; if the comprehensive confidence score is between the first threshold and the second threshold, then performing feature enhancement processing on the target frame hatch image, and determining the coordinate information of the hatch feature points based on the feature-enhanced target frame hatch image.

[0014] In some embodiments, the state equation used in the Kalman filter algorithm is: ;

[0015] in, Represents the position / velocity / acceleration state vector; Indicates the first The state of the frame; Indicates the first The state of the frame; This indicates process noise.

[0016] In some embodiments, the plurality of confidence assessment factors include: displacement consistency assessment factor, texture sharpness assessment factor, trajectory fit assessment factor, and historical stability assessment factor; wherein, fusing the plurality of confidence assessment factors to obtain a comprehensive confidence score includes: fusing the plurality of confidence assessment factors using the following formula: ; in, Indicates the overall confidence level; Indicates the displacement consistency assessment factor; This represents the texture sharpness evaluation factor; This represents the trajectory fit evaluation factor; This represents the historical stability assessment factor.

[0017] In some embodiments, the displacement consistency evaluation factor The texture sharpness evaluation factor The trajectory fit evaluation factor The historical stability assessment factors The calculation formula is as follows: ; ; ; ; in, Indicates the first Displacement deviation of each characteristic point; This represents the average displacement of all feature points; Indicates the number of displacement data points involved in the calculation; Represents the variance of the feature point displacement vector; Represents the image gradient; The Frobenius norm represents the gradient of an image; This represents the maximum value of the image gradient; Represents the residual sensitivity coefficient; This represents the residual vector of the trajectory fitting. Indicates maximum speed; This represents the standard deviation of velocity.

[0018] In some embodiments, the visual tracking algorithm is an improved Simultaneous Localization and Mapping (SLAM) algorithm. The improved SLAM algorithm converts the hatch images captured by the upper and lower binocular cameras into hatch images captured by the horizontal binocular camera by image transposition, and performs inter-frame registration using optical flow calculation / feature matching.

[0019] According to another aspect of this disclosure, a visual tracking device for a boarding bridge is also provided, comprising: a data acquisition module for acquiring multiple frames of cabin door images continuously acquired by a vision system mounted on the boarding bridge; a tracking window determination module for dynamically adjusting the size information of the tracking window in the target frame cabin door image based on the motion information of cabin door feature points in the target frame cabin door image, wherein the target frame cabin door image is any one of the multiple frames of cabin door images; and a visual tracking module for extracting output coordinate information of cabin door feature points from the tracking window of the target frame cabin door image based on a visual tracking algorithm.

[0020] According to another aspect of this disclosure, an electronic device is also provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the visual tracking method for a boarding bridge described in any of the preceding claims by executing the executable instructions.

[0021] According to another aspect of this disclosure, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the visual tracking method for boarding bridges as described in any of the preceding claims.

[0022] According to another aspect of this disclosure, a computer program product is also provided, comprising: a computer program or instructions that, when executed by a processor, implement the visual tracking method for a boarding bridge as described in any of the preceding claims.

[0023] The visual tracking method, apparatus, and related equipment for boarding bridges provided in this disclosure acquire multiple frames of cabin door images continuously collected by a vision system mounted on the boarding bridge; dynamically adjust the size information of the tracking window in the target frame cabin door image based on the motion information of cabin door feature points in any frame of the multiple cabin door images, and use the boarding bridge to extract cabin door feature points from the tracking window of the target frame cabin door image and output coordinate information.

[0024] By dynamically adjusting the size of the tracking window based on the motion information of the boarding bridge during the docking process with the aircraft door, the problem of losing feature points within the tracking window due to the movement of the boarding bridge can be avoided, thus improving the visual tracking effect.

[0025] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0026] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0027] Figure 1 This diagram illustrates an application system architecture according to an embodiment of the present disclosure. Figure 2 A flowchart of a visual tracking method for a boarding bridge is shown in an embodiment of this disclosure; Figure 3 This document illustrates a flowchart of an embodiment of visual tracking optimization. Figure 4 This diagram illustrates a tracking and positioning method using optical flow in an embodiment of the present disclosure. Figure 5 This diagram illustrates a method for fusing multiple confidence assessment factors according to an embodiment of the present disclosure. Figure 6 This diagram illustrates an embodiment of the present disclosure of optimizing visual tracking results based on comprehensive confidence. Figure 7 This diagram illustrates a visual tracking system architecture that utilizes optical flow for inter-frame registration according to an embodiment of the present disclosure. Figure 8 A schematic diagram of a visual tracking device for a boarding bridge is shown in an embodiment of this disclosure; Figure 9 A structural block diagram of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0028] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0029] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0030] To facilitate understanding, before introducing the embodiments of this disclosure, the following explanations are provided for several terms involved in the embodiments of this disclosure: SLAM: Simultaneous Localization and Mapping.

[0031] BRIEF: Binary Robust Independent Elementary Features, is a lightweight image local feature descriptor that represents the local gray-level distribution of feature points through binary strings. It has the advantages of fast generation and high storage / matching efficiency.

[0032] ORB: Oriented FAST and Rotated BRIEF.

[0033] The specific implementation methods of the embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0034] Figure 1 A schematic diagram of an exemplary application system architecture for a boarding bridge, applicable to embodiments of this disclosure, is shown. For example... Figure 1 As shown, the system architecture may include terminal device 10, network 20 and server 30.

[0035] Network 20 is a medium used to provide a communication link between terminal device 10 and server 30, and can be a wired network or a wireless network.

[0036] Optionally, the aforementioned wireless or wired networks use standard communication technologies and / or protocols. The network is typically the Internet, but can also be any network, including but not limited to Local Area Networks (LANs), Metropolitan Area Networks (MANs), Wide Area Networks (WANs), mobile, wired or wireless networks, private networks, or any combination of virtual private networks. In some embodiments, technologies and / or formats, including Hyper Text Markup Language (HTML), Extensible Markup Language (XML), etc., are used to represent data exchanged over the network. Furthermore, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Networks (VPNs), and Internet Protocol Security (IPSec) can be used to encrypt all or some links. In other embodiments, custom and / or dedicated data communication technologies can be used to replace or supplement the aforementioned data communication technologies.

[0037] In some embodiments of this disclosure, the terminal device 10 may be any electronic device, including but not limited to mobile phones, tablet computers, laptop computers, notebook computers, personal digital assistants (PDAs), handheld computers, netbooks, ultra-mobile personal computers (UMPCs), mobile internet devices (MIDs), augmented reality (AR) devices, virtual reality (VR) devices, robots, wearable devices, flight vehicles, vehicle user equipment (VUEs), shipboard devices, pedestrian user equipment (PUEs), smart home devices (home devices with wireless communication capabilities, such as refrigerators, televisions, washing machines, or furniture), game consoles, personal computers (PCs), ATMs, or self-service machines, etc. Wearable devices include: smartwatches, smart bracelets, smart earphones, smart glasses, smart jewelry (smart bracelets, smart chains, smart rings, smart necklaces, smart anklets, smart anklets, etc.), smart wristbands, smart clothing, etc. Among these, in-vehicle devices can also be referred to as in-vehicle terminals, in-vehicle controllers, in-vehicle modules, in-vehicle components, in-vehicle chips, or in-vehicle units, etc. It should be noted that the specific type of terminal device 10 is not limited in the embodiments disclosed herein.

[0038] Optionally, the client of the application installed on different terminal devices 10 may be the same, or the client of the same type of application based on different operating systems. Depending on the terminal platform, the specific form of the application client may also be different; for example, the application client may be a mobile client, a PC client, etc.

[0039] Server 30 can be a server that provides various services, such as a backend management server that supports the device operated by the user using terminal device 10. The backend management server can analyze and process received requests and other data, and feed the processing results back to the terminal device.

[0040] Optionally, the server can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0041] Those skilled in the art will know that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative; any number of terminal devices, networks, and servers can be included depending on actual needs. This disclosure does not limit the scope of the embodiments.

[0042] Under the above system architecture, this disclosure provides a visual tracking method for boarding bridges, which can be executed by any electronic device with computing power.

[0043] In some embodiments, the boarding bridge method provided in this disclosure can be executed by a terminal device in the above-described system architecture; in other embodiments, the boarding bridge method provided in this disclosure can be executed by a server in the above-described system architecture; in still other embodiments, the visual tracking method for boarding bridges provided in this disclosure can be implemented by the terminal device and the server in the above-described system architecture through interaction.

[0044] Figure 2 A flowchart of a method for a boarding bridge according to an embodiment of this disclosure is shown, such as Figure 2 As shown, the method includes the following steps: S202, acquires multiple frames of cabin door images continuously collected by the vision system mounted on the boarding bridge.

[0045] In this embodiment, a boarding bridge refers to an enclosed passageway connecting the terminal waiting area and the aircraft door, enabling barrier-free boarding / disembarking for passengers, crew members, and some cargo. Its body possesses degrees of freedom of movement, including extension, rotation, and pitch, and can adapt to different aircraft types and parking positions. The docking process between the boarding bridge and the door refers to controlling the boarding bridge to move from its initial parking position to the aircraft door and precisely engage with it. During this docking process, the boarding bridge moves to the aircraft door through alternating forward and rotational movements. The vision system is a visual perception device mounted on the front end of the boarding bridge, typically a camera or webcam, capable of capturing images of the aircraft door and surrounding area in real time. During the boarding bridge docking process, the vision system mounted on the boarding bridge can continuously capture multiple frames of images of the door and its surrounding environment at regular time intervals, obtaining multiple door images.

[0046] It should be noted that the visual tracking method provided in this disclosure can be applied to, but is not limited to, scenarios where boarding bridges dock with aircraft doors. In some embodiments, it can also be applied to scenarios such as boarding bridges docking with ship doors.

[0047] S204, dynamically adjust the size information of the tracking window in the target frame hatch image based on the motion information of the hatch feature points in the target frame hatch image, wherein the target frame hatch image is any frame hatch image among multiple frames of hatch images.

[0048] It should be noted that the target frame door image is any frame of the multi-frame door image, i.e., the door image currently being tracked and analyzed. During the docking process between the boarding bridge and the aircraft door, the size of the tracking window in the currently acquired door image is dynamically adjusted based on the motion information of the door feature points in the target frame door image at the current moment. The motion information of the door feature points includes, but is not limited to, the velocity, acceleration, and position of the door feature points. The tracking window is the area in the door image used to extract the tracking target (such as the aircraft door). This area is specifically used to lock onto and track the door target, and its function is to define the effective range of image analysis and avoid interference from irrelevant background (such as other parts of the fuselage, sky, and ground) with the door's positioning accuracy. If the tracking target is an aircraft door, the tracking window will usually cover the key feature areas of the door (such as the door frame, bottom corners, etc.). In a visual tracking system, if the tracking window size is too large, it will introduce more background noise; if the tracking window size is too small, it may lose some features of the door target, affecting tracking stability.

[0049] As the boarding bridge approaches or adjusts its attitude, the relative distance and angle between it and the aircraft door change. This is reflected in the image as a change in the pixel size and proportion of the door in the image. Therefore, in this embodiment of the present disclosure, during the process of the boarding bridge docking with the aircraft door, the motion information of the boarding bridge is indirectly reflected by the motion information of the door's feature points. The size information of the tracking window is dynamically adjusted to ensure that the vision system mounted on the boarding bridge can always accurately frame the door target.

[0050] In some embodiments, the motion information of the hatch feature point includes at least: the acceleration of the hatch feature point; the above S204 can be determined by the following formula. Size information of the tracking window in the frame hatch image: (1) in, Indicates the first The hatch feature points in the frame image are Pixel coordinates along the axis; Indicates the first The hatch feature points in the frame image are Pixel coordinates along the axis; Indicates the first Width of the tracking window in the frame hatch image (in pixels); Indicates the first The height of the tracking window in the frame hatch image (in pixels); and These are two parameters that change dynamically with time / frame and are related to the motion information of the hatch feature points.

[0051] Furthermore, in some embodiments, the motion information of the aforementioned hatch feature points includes at least: the velocity of the hatch feature points; and the... Width of the tracking window in the frame hatch image It can be calculated using the following formula: (2) (3) in, Indicates the baseline width of the tracking window; Indicates the reference height of the tracking window; Indicates the vibration sensitivity coefficient; A velocity vector representing the characteristics of the hatch; Represents a function related to the velocity of a characteristic point of the hatch; This indicates that the vision system is acquiring the first... The velocity vector of the hatch feature points in a frame of hatch image; This indicates that the vision system is acquiring the first... The velocity vector of the hatch feature points in a frame of hatch image; Indicates the first Frame and the The duration of the frame interval between frames; express and The magnitude of the difference vector is the change in velocity.

[0052] In some embodiments, the first The height of the tracking window in the frame hatch image It can be calculated using the following formula (4) or (5): (4) (5) in, Indicates whether the aspect ratio is fixed or variable; This indicates the reference height of the tracking window.

[0053] In this embodiment, the vibration sensitivity coefficient can range from 0.2 to 0.5, and the vibration sensitivity coefficient increases by 0.1 for every 1 Hz increase in vibration frequency. The vibration frequency here can be measured or calculated. For example, by tracking feature points in an image sequence and outputting the displacement data of the feature points in each frame (e.g., ...). Axial direction and The pixel offset in the axial direction is then converted into actual physical displacement, such as millimeters, through calibration. The displacement-time curve obtained by tracking (horizontal axis is time, vertical axis is displacement) is processed by signal processing (such as Fourier transform, power spectrum analysis), and the main frequency component of the curve is extracted to obtain the vibration frequency of the feature point.

[0054] The aforementioned tracking window is the search window for inter-frame registration in visual tracking. It can be used for feature point search in optical flow methods and for adjusting the search area in SLAM inter-frame feature matching (such as ORB features of hatch feature points). In visual tracking, ORB features refer to the combined features of "oriented FAST corner points + rotated BRIEF (Binary Robust Independent Elementary Features) descriptors" used for target tracking tasks. These are core local features for locating and matching targets in visual tracking algorithms. For example, if the target that visual tracking needs to continuously lock onto is a hatch, ORB features will first detect FAST corner points in the initial frame (quickly locating significant feature regions such as the edges and corners of the hatch), and then generate a BRIEF binary descriptor with rotation direction for each corner point, using this as the feature identifier of the target. The BRIEF binary descriptor is a lightweight method for describing local image features. Its core is to represent the local grayscale distribution features of feature points by generating binary strings, and it is commonly used in feature matching tasks in computer vision.

[0055] S206, based on a visual tracking algorithm, extracts hatch feature points from the tracking window of the target frame hatch image and outputs coordinate information.

[0056] It should be noted that the hatch feature points in this embodiment refer to feature points selected on the hatch that can determine the position of the hatch. These can be the outline of the hatch, the edge line of the hatch threshold, or the two corner points formed by the intersection of the edge lines on both sides of the hatch and the edge line of the threshold.

[0057] Taking an aircraft cabin door as an example, after the aircraft docks at the airport, the boarding bridge can use the left / right corner points of the cabin door as reference points to achieve precise docking and ensure safe passage for passengers. Therefore, the cabin door feature points in this embodiment include: a first corner point and a second corner point. The first corner point and the second corner point are the points formed by the intersection of the edge lines on both sides of the cabin door and the edge line of the door sill, respectively. When an observer outside the aircraft is facing the cabin door, the first corner point can be the left corner point of the cabin door, and the second corner point can be the right corner point of the cabin door.

[0058] In an optional embodiment, after outputting the coordinate information of the door feature points through the visual tracking method provided in this embodiment, the boarding bridge control system can determine the relative pose information between the door and the boarding bridge based on the coordinate information of the door feature points, and then control the boarding bridge to dock with the door based on the relative pose information.

[0059] In some embodiments, before extracting door feature points and outputting coordinate information from the tracking window of the target frame door image based on a visual tracking algorithm, the visual tracking method for boarding bridges provided in this disclosure may further include the following steps: S302, identify multiple confidence assessment factors for quantitative evaluation of visual tracking results; S304, integrates multiple confidence assessment factors to obtain a comprehensive confidence score; S306. Determine whether to optimize the visual tracking results based on the range of the overall confidence level.

[0060] It should be noted that visual tracking systems are susceptible to interference from complex environments in boarding bridge docking scenarios (such as strong airport light / backlight, rain and fog, aircraft reflection, and ground obstruction), which may lead to problems such as misidentification of feature points and tracking window offset, resulting in deviations in subsequent pose calculations. In this embodiment, multiple confidence evaluation factors are used to quantitatively assess the effectiveness of the tracking results from different perspectives. By fusing multiple confidence evaluation factors to obtain a comprehensive confidence score, low-reliability tracking results can be accurately identified (such as anomalies where a single factor meets the standard but the overall score of multiple factors is low), avoiding the transmission of incorrect feature point coordinates to subsequent docking control processes, and significantly reducing safety hazards such as boarding bridge offset and aircraft collisions caused by visual perception errors.

[0061] In some embodiments, determining whether to optimize the visual tracking result based on the overall confidence level includes: if the overall confidence level is less than a first threshold, then using a Kalman filter algorithm to predict the coordinate information of the hatch feature points in the target frame hatch image; if the overall confidence level is greater than a second threshold, then directly extracting the coordinate information of the hatch feature points from the tracking window of the target frame hatch image based on the visual tracking algorithm, wherein the second threshold is greater than the first threshold; if the overall confidence level is between the first threshold and the second threshold, then performing feature enhancement processing on the target frame hatch image, and determining the coordinate information of the hatch feature points based on the feature-enhanced target frame hatch image.

[0062] When the overall confidence score is less than the first threshold, it indicates that the visual tracking result is very poor (e.g., the hatch feature points are completely occluded, or the image is severely blurred, making the feature points unrecognizable). In this case, the visual data of the current frame is no longer relied upon; instead, the coordinates of the hatch feature points are predicted using the Kalman filter algorithm. Since the core of the Kalman filter is to combine the historical motion information of the boarding bridge (such as the coordinates of feature points in previous frames, the boarding bridge's velocity / acceleration) and the system motion model to estimate the coordinates of the hatch feature points in the current frame, it can fill the gaps in visual tracking and prevent docking control interruptions due to the loss of feature points.

[0063] When the overall confidence score is between the first and second thresholds, it indicates that the visual tracking result has some noise but is not completely invalid (e.g., uneven lighting causes blurred feature point edges, slight occlusion causes partial feature loss). In this case, feature enhancement processing is first performed on the target frame hatch image, and then the coordinates are re-extracted. In some embodiments, the feature enhancement method may include, but is not limited to: ① Image preprocessing: Histogram equalization is used to improve the uneven lighting problem, and Gaussian filtering is used to eliminate noise; ② Edge enhancement: The contrast of the hatch edge line is enhanced by the Sobel operator and the Laplacian operator to highlight corner features; ③ Local region repair: Interpolation is performed on slightly occluded areas to restore the integrity of the hatch features; ④ Feature points are extracted again after enhancement, which can effectively improve the accuracy of coordinates.

[0064] When the overall confidence level is greater than the second threshold, it indicates that the visual tracking result is highly reliable (clear image, good feature point matching, and strong inter-frame stability). At this time, no additional processing is required. The coordinate information of the hatch feature points can be directly extracted from the tracking window, which not only ensures the real-time performance of the data but also avoids the impact of redundant operations on docking control efficiency.

[0065] In this embodiment, a hierarchical scheme is designed for tracking results with different confidence levels. When the overall confidence level is less than a first threshold, prediction padding is used to buy time for the system to resume visual tracking and avoid docking process stalls. When the overall confidence level is between the first and second thresholds, feature enhancement is used to repair the hatch image, correcting data deviations without sacrificing real-time performance, balancing accuracy and efficiency. When the overall confidence level is greater than the second threshold, the visual tracking result is directly adopted, eliminating optimization steps and ensuring the efficiency of the docking process. This embodiment can solve the problem of feature point loss in extreme environments using a single visual algorithm.

[0066] In some embodiments, the multiple confidence assessment factors include: displacement consistency assessment factor, texture sharpness assessment factor, trajectory fit assessment factor, and historical stability assessment factor; wherein, fusing multiple confidence assessment factors to obtain a comprehensive confidence score includes: fusing multiple confidence assessment factors using the following formula: (6) in, Indicates the overall confidence level; Indicates the displacement consistency assessment factor; This represents the texture sharpness evaluation factor; This represents the trajectory fit evaluation factor; This represents the historical stability assessment factor.

[0067] Furthermore, in some embodiments, the displacement consistency evaluation factor Texture sharpness evaluation factor Trajectory fit evaluation factor Historical stability assessment factors The calculation formula is as follows: (7) (8) (9) (10) in, Indicates the first The displacement deviation of the nth feature point (i.e., the nth feature point) The actual displacement vector of each feature point and (deviation) This represents the average displacement of all feature points (reflecting the overall movement trend of the feature point group); Indicates the number of displacement data points involved in the calculation; Represents the variance of the feature point displacement vector; Represents the image gradient; The Frobenius norm represents the gradient of an image; This represents the maximum value of the image gradient; Represents the residual sensitivity coefficient; This represents the residual vector of the trajectory fitting. This indicates the maximum speed (the system allows a maximum speed of 0.5 m / s). This represents the standard deviation of velocity.

[0068] Displacement consistency evaluation factor Used to evaluate the motion consistency of feature groups, reflecting the characteristics of rigid body motion. As a rigid body, all feature points should have the same motion vector. If some points deviate abnormally, the tracking reliability decreases; displacement consistency evaluation factor. The value range is [0,1], and a value of 1 indicates that they are completely identical.

[0069] Texture sharpness evaluation factor Used to quantify the discriminative power of hatch edge features and resist rain / fog / blur interference; hatch tracking relies on high-contrast edges (such as hatch seams); rain and fog can cause edge blurring, leading to decreased tracking accuracy; texture sharpness evaluation factor. The value range is [0,1], with a value of 1 indicating the clearest edge. The calculation process is as follows: ① Calculate the Sobel gradient for the ROI region; ② Calculate the F-norm of the gradient matrix; ③ Normalize by dividing by the maximum gradient value of the entire image.

[0070] Trajectory fit evaluation factor Used to assess the degree to which a motion trajectory conforms to physical laws, the docking process of boarding bridges typically involves low-speed linear motion. A sudden change in the trajectory could lead to mistracking. In one embodiment, the residual sensitivity coefficient... The calculation formula is as follows: (11) The residual sensitivity coefficient is negatively correlated with the velocity of the feature point, strengthening trajectory constraints at low speeds. This residual sensitivity coefficient is also called the adaptive sensitivity coefficient. When moving at a constant speed, the smaller the speed, the larger the adaptive sensitivity coefficient, strictly controlling trajectory smoothness; when moving at varying speeds, the larger the speed, the smaller the adaptive sensitivity coefficient, allowing trajectory changes.

[0071] Historical stability assessment factor This method is used to quantify long-term motion stability and detect cumulative errors. Small velocity fluctuations over multiple consecutive frames indicate a stable system. Sudden velocity changes may indicate tracking drift. The implementation method is as follows: ① Store the velocity sequence of the most recent 10 frames; ② Calculate the standard deviation. ③ Use Normalization yields a stability index.

[0072] In this embodiment, the overall confidence level is the product of multiple confidence assessment factors. A high overall confidence level is achieved only when the values ​​of all confidence assessment factors across all dimensions are relatively large, avoiding situations where some confidence assessment factors have low values ​​but the overall confidence level is high. The overall confidence level in the above embodiment more accurately reflects the overall reliability of the visual tracking results. During boarding bridge docking, visual tracking faces dynamically changing interferences (such as image magnification leading to texture changes when the boarding bridge approaches the cabin door, or slight aircraft shaking causing fluctuations in feature point displacement). The four confidence assessment factors proposed in this embodiment can basically cover possible dynamic scenarios. For example, the displacement consistency assessment factor can handle sudden feature point shifts; the texture clarity assessment factor can handle quality degradation caused by image scaling and lighting changes; the trajectory fitting degree assessment factor can handle jumps in time-series tracking; and the historical stability assessment factor can handle drift during long-term motion. Through these four assessment factors, this embodiment enables the confidence assessment to accurately match the actual needs of boarding bridge docking, ensuring the reliability of visual tracking in dynamic scenarios.

[0073] Optical flow computation can quickly track the pixel displacement of feature points in adjacent frames, offering high computational efficiency and meeting the real-time requirements of boarding bridge docking (adapting to the dynamic movement rhythm of the boarding bridge). Feature matching, through global matching of feature points such as SIFT / SURF, can provide inter-frame correlation when optical flow tracking fails (e.g., feature point occlusion), improving the robustness of inter-frame registration. In one embodiment, an inter-frame registration method combining optical flow computation and feature matching can ensure the real-time response speed of the SLAM algorithm while also handling interference from local occlusion of the door image and changes in illumination, allowing the SLAM algorithm to operate stably in the complex dynamic scene of boarding bridge docking.

[0074] The visual tracking method for boarding bridges provided in this disclosure can be applied, but is not limited to, real-time tracking of door targets during aircraft landing and inter-frame registration of SLAM visual odometry (VO) in boarding bridge docking scenarios, and is particularly suitable for pose estimation and status monitoring of door targets.

[0075] Figure 4 This diagram illustrates a tracking and positioning process using optical flow in an embodiment of the present disclosure, as shown below. Figure 4 As shown, the input is an image sequence obtained by the vision system continuously acquiring images of the hatch, and the output is the position of feature points (such as left and right corner points) in the hatch images in the two consecutive frames of hatch images.

[0076] Assuming the hatch feature point is at The location in the frame hatch image is Traditional optical flow tracing methods use a fixed-size tracking window, so the first... The size information of the tracking window in the frame hatch image is ,in, and The value is a constant; in this embodiment of the disclosure, a dynamically adjusted tracking window is used, then the first... The size information of the tracking window in the frame hatch image is .

[0077] In this embodiment, the size information of the tracking window adaptively expands with the acceleration of the feature point movement, which can improve the success rate of optical flow tracking in vibration scenarios and solve the problem of feature matching loss caused by vibration in SLAM inter-frame registration. It can be adapted to the SLAM visual odometry requirements of boarding bridge docking scenarios.

[0078] Existing SLAM frameworks (such as ORB-SLAM2) rely on a fixed search window and a uniform motion prior for continuous inter-frame registration. In boarding bridge docking scenarios, vibration can easily lead to the loss of SLAM feature point matching, and the boarding bridge's rotation-forward alternating control mode can cause the uniform motion prior to fail. In this embodiment, the size information of the tracking window adaptively expands with the acceleration of the boarding bridge's motion, which can effectively improve the success rate of feature point tracking in vibration scenarios.

[0079] Currently, the effectiveness of visual tracking results is evaluated through manual monitoring, lacking quantitative assessment of visual tracking results, making it difficult to achieve early fault warning.

[0080] To achieve a quantitative evaluation of visual tracking results, the embodiments of this disclosure propose... Figure 5 The four-position confidence assessment model shown can be used not only for the health quantification of optical flow tracking, but also as a tracking status assessment index for SLAM visual odometry (VO). Specifically, the displacement consistency assessment factor is used to judge the rigid body motion characteristics of SLAM feature matching, the trajectory fitting degree assessment factor is used to constrain the rationality of SLAM motion model, and the historical stability assessment factor is used to suppress SLAM cumulative drift, providing a quantitative basis for early warning and graded response to SLAM tracking failure.

[0081] Figure 6 The illustration shows a schematic diagram of optimizing visual tracking results based on comprehensive confidence in an embodiment of this disclosure, such as... Figure 6 As shown, when the overall confidence level is greater than the second threshold (e.g., 0.85), normal tracking is performed, and the coordinate information obtained from the tracking is directly output; when the overall confidence level is between the first threshold (0.6) and the second threshold (0.85), local features are enhanced, and SURF feature enhancement is activated; when the overall confidence level is less than the first threshold (0.6), re-detection is guided based on kinematic prediction, and Kalman prediction is performed based on historical trajectories.

[0082] For example, Figure 7 This illustration shows a schematic diagram of a visual tracking system architecture that utilizes optical flow for inter-frame registration, as described in an embodiment of this disclosure. Figure 7 As shown, the image acquisition module continuously acquires multiple frames of cabin door images; the optical flow tracing module calculates the motion information of cabin door feature points based on the acquired multiple frames of cabin door images; the confidence assessment module evaluates the optical flow tracing results; the response decision module determines the corresponding processing measures based on the confidence assessment results; and the control execution module executes the docking control of the boarding bridge according to the instructions of the response decision module. The core algorithm flow mainly includes: (a) Optical flow calculation: Input: Images of adjacent frames acquired by the image acquisition module; Solve the equation as follows: (12) in, Indicates the image in Spatial gradient along the axis; Indicates the image in Spatial gradient along the axis; Indicates feature points at Displacement in the axial direction; Indicates feature points at Displacement in the axial direction; optical flow vector Indicates feature points from the th Frame to the The direction and distance of the frame's motion; This refers to the rate of change of brightness at the same pixel location over time. The window size selected for optical flow calculation can be determined by formula (2) above to determine the width information, and by formula (4) or (5) above to determine the height information.

[0083] (II) Confidence Assessment Process: For each feature point, the displacement is calculated using the following formula: (13) in, Indicates the first The displacement deviation of each hatch feature point (such as the left / right corner of the hatch); Indicates the first Coordinate information of each hatch feature point in the current frame hatch image; Indicates the pre-calibrated first Coordinate information of each hatch feature point.

[0084] After calculating the local gradient, the trajectory fitting residual is updated, and the historical velocity sequence is calculated. The confidence evaluation factors of the four dimensions are calculated by the above formulas (7) to (10), and the comprehensive confidence is calculated by the above formula (6).

[0085] (III) Kalman Filter Algorithm Prediction: In this embodiment of the disclosure, the state equation used by the Kalman filter algorithm is as follows: (14) (15) in, Represents the position / velocity / acceleration state vector. For location, For speed, For acceleration; Indicates the first The frame's position / velocity / acceleration state vector; Indicates the first The frame's position / velocity / acceleration state vector; Indicates a time interval; The process noise represents disturbances not considered by the model (such as minor vibrations of the boarding bridge or sudden shifts in feature points), and is a random vector following a Gaussian distribution. The covariance of the process noise... It is expressed as follows: (16) in, The fundamental covariance is a pre-set initial value that represents the noise level when there is no interference. This indicates the overall confidence level. The larger the value of (the more reliable the visual tracking results), the greater the covariance of the process noise. The smaller the value, the lower the weight of the model's predictions, and the more it relies on the visual tracking results; conversely, the higher the overall confidence level. The smaller the value of , the lower the covariance of the process noise. The larger the value, the higher the weight of the model's predictions.

[0086] As an example, assuming the current scenario is a strong crosswind (wind speed 12 m / s), this scenario affects the confidence assessment factors in the following two dimensions: displacement consistency assessment factor. Decline (feature points affected by wind disturbance); trajectory fit evaluation factor decline.

[0087] System Response: Enhancing Historical Stability Assessment Factors The weight is set to 0.4; the window size is automatically increased by 15%; and the feature enhancement mode is switched.

[0088] Experimental results: Tracking accuracy remained <2cm.

[0089] In one embodiment, common parameter configurations are shown in Table 1.

[0090] Table 1

[0091] As can be seen from the above, the visual tracking method for boarding bridges provided in this embodiment dynamically adjusts the search window size for visual frame registration based on the motion state (such as acceleration and velocity) of feature points; jointly evaluates the confidence of visual tracking / registration through four confidence evaluation factors: displacement consistency, texture clarity, trajectory fitting degree, and historical stability; evaluates the visual tracking results quantitatively based on the comprehensive confidence metric; and performs feature enhancement processing or obtains the coordinate information of feature points based on motion prediction when the reliability of the visual tracking results is not high.

[0092] It should be noted that the acquisition, storage, use, and processing of data in this disclosed technical solution comply with the relevant provisions of laws and regulations. All types of data, such as personal identity data, operational data, and behavioral data related to individuals, customers, and groups, obtained in this disclosed embodiment have been agreed upon by the users.

[0093] Based on the same inventive concept, this disclosure also provides a visual tracking device for boarding bridges, as described in the following embodiments. Since the principle by which this device embodiment solves the problem is similar to that of the above-described method embodiments, the implementation of this device embodiment can refer to the implementation of the above-described method embodiments, and repeated details will not be elaborated further.

[0094] Figure 8 This illustration shows a schematic diagram of a visual tracking device for a boarding bridge according to an embodiment of the present disclosure, such as... Figure 8 As shown, the device includes: a data acquisition module 801, a tracking window determination module 802, and a visual tracking module 803.

[0095] The data acquisition module 801 is used to acquire multiple frames of cabin door images continuously acquired by the vision system mounted on the boarding bridge; the tracking window determination module 802 is used to dynamically adjust the size information of the tracking window in the target frame cabin door image according to the motion information of the cabin door feature points in the target frame cabin door image, wherein the target frame cabin door image is any frame cabin door image among the multiple frames cabin door images; the visual tracking module 803 is used to extract cabin door feature points from the tracking window of the target frame cabin door image based on the visual tracking algorithm and output the coordinate information.

[0096] In some embodiments, the motion information of the hatch feature point includes at least the acceleration of the hatch feature point; then the above-mentioned tracking window determination module is also used to determine the width information of the tracking window by the above-mentioned formula (2), and to determine the height information of the tracking window by the above-mentioned formula (4) or (5).

[0097] In an optional embodiment, after the visual tracking device provided in this embodiment outputs the coordinate information of the door feature points, the boarding bridge control system can determine the relative pose information between the door and the boarding bridge based on the coordinate information of the door feature points, and then control the boarding bridge to dock with the door based on the relative pose information.

[0098] In some embodiments, the relative pose information includes: angular deviation and / or positional deviation; wherein, controlling the boarding bridge to dock with the cabin door based on the relative pose information includes: controlling the boarding bridge to perform rotation and / or forward movement based on the angular deviation and / or positional deviation.

[0099] In some embodiments, the visual tracking device for boarding bridges provided in this disclosure further includes: a confidence assessment factor determination module 804, used to determine multiple confidence assessment factors for quantitatively evaluating the visual tracking results; a comprehensive confidence determination module 805, used to fuse the multiple confidence assessment factors to obtain a comprehensive confidence level; and a visual tracking result optimization module 806, used to determine whether to optimize the visual tracking results based on the range of values ​​of the comprehensive confidence level.

[0100] In some embodiments, the visual tracking result optimization module 806 is further configured to: if the comprehensive confidence level is less than a first threshold, use a Kalman filter algorithm to predict the coordinate information of the hatch feature points in the target frame hatch image; if the comprehensive confidence level is greater than a second threshold, directly extract the coordinate information of the hatch feature points from the tracking window of the target frame hatch image based on the visual tracking algorithm, wherein the second threshold is greater than the first threshold; if the comprehensive confidence level is between the first threshold and the second threshold, perform feature enhancement processing on the target frame hatch image, and determine the coordinate information of the hatch feature points based on the feature-enhanced target frame hatch image.

[0101] In some embodiments, the multiple confidence assessment factors include: displacement consistency assessment factor, texture clarity assessment factor, trajectory fit assessment factor, and historical stability assessment factor; the above-mentioned comprehensive confidence determination module is also used to fuse the multiple confidence assessment factors through the above formula (6).

[0102] In some embodiments, the visual tracking algorithm is an improved Simultaneous Localization and Mapping (SLAM) algorithm. The improved SLAM algorithm converts the hatch images captured by the upper and lower binocular cameras into hatch images captured by the horizontal binocular camera by image transposition, and performs inter-frame registration using optical flow calculation / feature matching.

[0103] It should be noted that the examples and application scenarios implemented by the modules in the above device embodiments and the corresponding steps in the method embodiments are the same, but are not limited to the content disclosed in the above method embodiments. It should also be noted that the above modules, as part of the device, can be executed in a computer system such as a set of computer-executable instructions.

[0104] Those skilled in the art will understand that various aspects of this disclosure can be implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which can be collectively referred to herein as a "circuit", "module" or "system".

[0105] Based on the same inventive concept, this disclosure also provides an electronic device, which includes: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the visual tracking method for a boarding bridge described above by executing the executable instructions. Since the principle by which this electronic device embodiment solves the problem is similar to that of the above method embodiments, the implementation of this electronic device embodiment can refer to the implementation of the above method embodiments, and repeated details will not be described again.

[0106] The following reference Figure 9 To describe an electronic device 900 according to such an embodiment of the present disclosure. Figure 9 The electronic device 900 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0107] like Figure 9 As shown, the electronic device 900 is manifested in the form of a general-purpose computing device. The components of the electronic device 900 may include, but are not limited to: at least one processing unit 910, at least one storage unit 920, and a bus 930 connecting different system components (including storage unit 920 and processing unit 910).

[0108] The storage unit stores program code that can be executed by the processing unit 910, causing the processing unit 910 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure. For example, the processing unit 910 can perform the following steps of the above method embodiment: acquiring multiple frames of cabin door images continuously collected by a vision system mounted on the boarding bridge; dynamically adjusting the size information of the tracking window in the target frame cabin door image based on the motion information of the cabin door feature points in the target frame cabin door image, wherein the target frame cabin door image is any frame of the multiple cabin door images; and extracting cabin door feature points from the tracking window of the target frame cabin door image and outputting coordinate information based on a visual tracking algorithm.

[0109] Storage unit 920 may include readable media in the form of volatile storage units, such as random access memory (RAM) 9201 and / or cache memory 9202, and may further include read-only memory (ROM) 9203.

[0110] The storage unit 920 may also include a program / utility 9204 having a set (at least one) program module 9205, such program module 9205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0111] Bus 930 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0112] Electronic device 900 can also communicate with one or more external devices 940 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 900, and / or with any device that enables electronic device 900 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 950. Furthermore, electronic device 900 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 960. As shown, network adapter 960 communicates with other modules of electronic device 900 via bus 930. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 900, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0113] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0114] Based on the same inventive concept, this disclosure also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements any of the above-described visual tracking methods for boarding bridges. Since the principle by which this computer-readable storage medium embodiment solves the problem is similar to that of the above-described method embodiments, the implementation of this computer-readable storage medium embodiment can refer to the implementation of the above-described method embodiments, and repeated details will not be elaborated further.

[0115] More specific examples of computer-readable storage media in this disclosure may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0116] In this disclosure, a computer-readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of transmitting, propagating, or transmitting a program for use by or in connection with an instruction execution system, apparatus, or device.

[0117] Optionally, the program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0118] In practical implementation, program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0119] Based on the same inventive concept, this disclosure also provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the visual tracking method for a boarding bridge according to any one of the above method embodiments. Since the principle by which this computer program product embodiment solves the problem is similar to that of the above method embodiments, the implementation of this computer program product embodiment can refer to the implementation of the above method embodiments, and repeated details will not be elaborated further.

[0120] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0121] Furthermore, although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0122] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0123] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.

Claims

1. A visual tracking method for boarding bridges, characterized in that, include: Acquire multiple frames of cabin door images continuously captured by a vision system mounted on the boarding bridge; Based on the motion information of the hatch feature points in the target frame hatch image, the size information of the tracking window in the target frame hatch image is dynamically adjusted, wherein the target frame hatch image is any frame hatch image in the multi-frame hatch images; Based on a visual tracking algorithm, the door feature points are extracted from the tracking window of the target frame door image and their coordinate information is output.

2. The visual tracking method for boarding bridges according to claim 1, characterized in that, Based on the motion information of the hatch feature points in the target frame hatch image, the size information of the tracking window in the target frame hatch image is dynamically adjusted, including: The number is determined by the following formula. Size information of the tracking window in the frame hatch image: ； in, Indicates the first The hatch feature points in the frame image are Pixel coordinates along the axis; Indicates the first The hatch feature points in the frame image are Pixel coordinates along the axis; Indicates the first The width of the tracking window in the frame hatch image; Indicates the first The height of the tracking window in the frame hatch image; and These are two parameters that change dynamically with time / frame and are related to the motion information of the hatch feature points.

3. The visual tracking method for boarding bridges according to claim 2, characterized in that, The motion information of the hatch feature points includes at least: the velocity of the hatch feature points; in, The calculation formula is as follows: ； ； in, Indicates the baseline width of the tracking window; Indicates the vibration sensitivity coefficient; A velocity vector representing the characteristics of the hatch; Represents a function related to the velocity of a characteristic point of the hatch; This indicates that the vision system is acquiring the first... The velocity vector of the hatch feature points in a frame of hatch image; This indicates that the vision system is acquiring the first... The velocity vector of the hatch feature points in a frame of hatch image; Indicates the first Frame and the The duration of the frame interval between frames; express and The magnitude of the difference vector is the change in velocity.

4. The visual tracking method for boarding bridges according to claim 3, characterized in that, The calculation formula is as follows: ； or, ; in, Indicates whether the aspect ratio is fixed or variable; This indicates the reference height of the tracking window.

5. The visual tracking method for boarding bridges according to claim 1, characterized in that, Before extracting hatch feature points and outputting coordinate information from the tracking window of the target frame hatch image based on a visual tracking algorithm, the method further includes: Identify multiple confidence assessment factors for quantifying visual tracking results; The multiple confidence assessment factors are fused to obtain the comprehensive confidence score. Based on the range of the comprehensive confidence level, determine whether to optimize the visual tracking results.

6. The visual tracking method for boarding bridges according to claim 5, characterized in that, Based on the comprehensive confidence level, determine whether to optimize the visual tracking results, including: If the overall confidence level is less than the first threshold, the Kalman filter algorithm is used to predict the coordinate information of the hatch feature points in the target frame hatch image. If the overall confidence level is greater than the second threshold, then the coordinate information of the hatch feature points is extracted directly from the tracking window of the target frame hatch image based on the visual tracking algorithm, where the second threshold is greater than the first threshold. If the overall confidence level is between the first threshold and the second threshold, feature enhancement processing is performed on the target frame hatch image, and the coordinate information of the hatch feature points is determined based on the feature-enhanced target frame hatch image.

7. The visual tracking method for boarding bridges according to claim 6, characterized in that, The state equation used in the Kalman filter algorithm is: ； in, Represents the position / velocity / acceleration state vector; Indicates the first The state of the frame; Indicates the first The state of the frame; This indicates process noise.

8. The visual tracking method for boarding bridges according to claim 5, characterized in that, The multiple confidence assessment factors include: displacement consistency assessment factor, texture sharpness assessment factor, trajectory fit assessment factor, and historical stability assessment factor; The method for fusing the multiple confidence assessment factors to obtain a comprehensive confidence score includes fusing the multiple confidence assessment factors using the following formula: ； in, Indicates the overall confidence level; Indicates the displacement consistency assessment factor; This represents the texture sharpness evaluation factor; This represents the trajectory fit evaluation factor; This represents the historical stability assessment factor.

9. The visual tracking method for boarding bridges according to claim 8, characterized in that, The displacement consistency evaluation factor The texture sharpness evaluation factor The trajectory fit evaluation factor The historical stability assessment factors The calculation formula is as follows: ； ； ； ； in, Indicates the first Displacement deviation of each characteristic point; This represents the average displacement of all feature points; Indicates the number of displacement data points involved in the calculation; Represents the variance of the feature point displacement vector; Represents the image gradient; The Frobenius norm represents the gradient of an image; This represents the maximum value of the image gradient; Represents the residual sensitivity coefficient; This represents the residual vector of the trajectory fitting. Indicates maximum speed; This represents the standard deviation of velocity.

10. A visual tracking device for an airplane boarding bridge, characterized in that, include: The data acquisition module is used to acquire multiple frames of cabin door images continuously collected by the vision system mounted on the boarding bridge; The tracking window determination module is used to dynamically adjust the size information of the tracking window in the target frame hatch image based on the motion information of the hatch feature points in the target frame hatch image, wherein the target frame hatch image is any one of the multiple frame hatch images; The visual tracking module is used to extract hatch feature points from the tracking window of the target frame hatch image and output coordinate information based on the visual tracking algorithm.

11. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the visual tracking method for a boarding bridge according to any one of claims 1 to 9 by executing the executable instructions.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the visual tracking method for boarding bridges as described in any one of claims 1 to 9.

13. A computer program product comprising: A computer program or instruction, characterized in that, when executed by a processor, the computer program or instruction implements the visual tracking method for boarding bridges as described in any one of claims 1 to 9.

Images