Target tracking method and system based on any binocular image and related equipment

Abstract

The invention provides a target tracking method and system based on any binocular image and related equipment, and relates to the technical field of computer vision. The method comprises the following steps: acquiring a first image and a second image of a current frame obtained by synchronously shooting a target object by a first visual device and a second visual device; performing image transposition processing on the first image and the second image to obtain a pseudo-horizontal binocular image; inputting the pseudo horizontal binocular image into a synchronous localization and mapping (SLAM) algorithm tracking module, and outputting the current pose of the target object in a visual coordinate system; calculating a relative pose intermediate quantity of the target object under the visual coordinate system according to the current pose and the key frame pose under the visual coordinate system; and performing axis unification processing on the relative pose intermediate quantity to output a relative pose final quantity under the control coordinate system. According to the method and the device, binocular vision equipment in arbitrary arrangement can multiplex a mature horizontal binocular SLAM tracking algorithm, and the hardware adaptation range of a binocular target tracking technology is greatly expanded.

Description

Technical Field

[0001] This disclosure relates to the field of computer vision technology, and in particular to a target tracking method, system and related equipment based on arbitrary binocular images. Background Technology

[0002] Traditional SLAM (Simultaneous Localization and Mapping) tracking algorithms are based on a horizontal baseline and a left-right camera layout. However, in the application scenario of boarding bridge docking doors, due to the limitation of the installation space of boarding and docking equipment, the binocular cameras are usually used in a compact structure of stacked vertically, which causes the binocular epipolar lines to change from the traditional near-horizontal direction to a near-vertical direction. Directly using the horizontal baseline SLAM tracking algorithm for door tracking will result in a large tracking error, thus affecting the accurate docking of the boarding bridge.

[0003] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0004] This disclosure provides a target tracking method, system, and related equipment based on arbitrary binocular images, which at least to some extent overcomes the technical problem in related technologies that target tracking schemes based on SLAM algorithms are difficult to adapt to both upper and lower binocular images.

[0005] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part by practice of this disclosure.

[0006] According to one aspect of this disclosure, a target tracking method based on arbitrary binocular images is provided, comprising: acquiring a first image and a second image of a current frame obtained by a first vision device and a second vision device simultaneously capturing images of a target object; performing image transpose processing on the first image and the second image to obtain a pseudo-horizontal binocular image, wherein the binocular baseline corresponding to the pseudo-horizontal binocular image is in the horizontal direction; inputting the pseudo-horizontal binocular image to a Simultaneous Localization and Mapping (SLAM) algorithm tracking module to output the current pose of the target object in the visual coordinate system; calculating the relative pose intermediate value of the target object in the visual coordinate system based on the current pose in the visual coordinate system and the keyframe pose; and performing axis-consistent processing on the relative pose intermediate value to output the final relative pose value in the control coordinate system.

[0007] In some embodiments, before performing image transpose processing on the first image and the second image, the method further includes: retrieving calibration intrinsic parameters and calibration extrinsic parameters, and performing image distortion correction and / or epipolar correction on the first image and the second image, respectively.

[0008] In some embodiments, performing image transpose processing on the first image and the second image to obtain a pseudo-horizontal binocular image includes: obtaining the angle between the binocular baselines of the first vision device and the second vision device relative to the horizontal plane; if the angle is 90°, then performing matrix transpose on the first image and the second image respectively, so that the original pixel column direction is mapped to the pixel row direction and the original pixel row direction is mapped to the pixel column direction, to obtain a pseudo-horizontal binocular image; if the angle is any other angle besides 0° and 90°, then performing rotation transpose on the first image and the second image respectively according to the angle to obtain a pseudo-horizontal binocular image.

[0009] In some embodiments, before inputting the pseudo-horizontal stereo image into the Simultaneous Localization and Mapping (SLAM) algorithm tracking module, the method further includes: downsampling the pseudo-horizontal stereo image; and / or masking the pseudo-horizontal stereo image to retain the pixel region where the target object is located.

[0010] In some embodiments, the target object is an aircraft door, and the first vision device and the second vision device are binocular vision devices mounted on a boarding bridge. The method further includes: detecting anchor points on the aircraft door frame in the keyframe image and obtaining the keyframe anchor point coordinates; transforming the keyframe anchor point coordinates to the current frame anchor point coordinate intermediate value through the relative pose intermediate value; reading calibration intrinsic parameters and calibration extrinsic parameters, and performing axis-consistent processing on the current frame anchor point coordinate intermediate value to output the final value of the current frame anchor point coordinates in the control coordinate system.

[0011] In some embodiments, the method further includes: reading calibration intrinsic parameters and calibration extrinsic parameters, initializing the synchronous localization and mapping (SLAM) algorithm tracking module and disabling loop closure detection.

[0012] According to one aspect of this disclosure, a target tracking device based on arbitrary binocular images is also provided, comprising: an image acquisition module for acquiring a first image and a second image of a current frame obtained by a first vision device and a second vision device simultaneously capturing images of a target object; an image transpose processing module for performing image transpose processing on the first image and the second image to obtain a pseudo-horizontal binocular image, wherein the binocular baseline corresponding to the pseudo-horizontal binocular image is in the horizontal direction; a SLAM tracking module for inputting the pseudo-horizontal binocular image to a Simultaneous Localization and Mapping (SLAM) algorithm tracking module and outputting the current pose of the target object in a visual coordinate system; a relative pose intermediate quantity determination module for calculating the relative pose intermediate quantity of the target object in a visual coordinate system based on the current pose and keyframe pose in the visual coordinate system; and a relative pose final quantity determination module for performing axis-consistent processing on the relative pose intermediate quantity to output the relative pose final quantity in a control coordinate system.

[0013] According to one aspect of this disclosure, a target tracking system based on arbitrary binocular images is also provided, comprising: a first vision device, a second vision device, and an image processing device; wherein, the first vision device is used to capture a first image of a target object; the second vision device is used to capture a second image of the target object; the image processing device is used to acquire the first image and the second image of the current frame obtained by the first vision device and the second vision device simultaneously capturing the target object; perform image transpose processing on the first image and the second image to obtain a pseudo-horizontal binocular image, wherein the binocular baseline corresponding to the pseudo-horizontal binocular image is in the horizontal direction; input the pseudo-horizontal binocular image to a Simultaneous Localization and Mapping (SLAM) algorithm tracking module, and output the current pose of the target object in the visual coordinate system; calculate the relative pose intermediate value of the target object in the visual coordinate system based on the current pose in the visual coordinate system and the keyframe pose; perform axis-consistent processing on the relative pose intermediate value to output the final relative pose value in the control coordinate system.

[0014] According to another aspect of this disclosure, an electronic device is also provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the target tracking method based on any of the preceding claims via executing the executable instructions.

[0015] According to another aspect of this disclosure, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the target tracking method based on any of the preceding claims.

[0016] According to another aspect of this disclosure, a computer program product is also provided, comprising: a computer program or instructions that, when executed by a processor, implement the target tracking method based on any of the preceding claims.

[0017] The target tracking method, system, and related devices based on arbitrary binocular images provided in the embodiments of this disclosure transpose the images acquired by the binocular vision device according to the calibration parameters of the binocular vision device to obtain a pseudo-horizontal binocular image with a horizontal baseline. The equivalent horizontal disparity of the pseudo-horizontal binocular image is calculated and input into the SLAM algorithm tracking module to output the position information of the target object in the spatial coordinate system. This disclosure converts binocular images with arbitrary baselines into pseudo-horizontal binocular images with equivalent horizontal disparity, allowing any arrangement of binocular vision devices to reuse mature horizontal binocular SLAM tracking algorithms. This significantly expands the hardware adaptability of binocular target tracking technology and solves the problem that traditional binocular SLAM tracking algorithms are only compatible with horizontal baseline binocular vision.

[0018] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0020] Figure 1 This diagram illustrates a target tracking system architecture based on arbitrary binocular images according to an embodiment of the present disclosure. Figure 2 This diagram illustrates a target tracking method based on arbitrary binocular images according to an embodiment of the present disclosure. Figure 3 This diagram illustrates a flowchart of a binocular image transposition process according to an embodiment of the present disclosure. Figure 4 This diagram illustrates a specific implementation method for target tracking in an embodiment of the present disclosure. Figure 5 This diagram illustrates an improved SLAM algorithm according to an embodiment of the present disclosure. Figure 6 This diagram illustrates a target tracking device based on arbitrary binocular images according to an embodiment of the present disclosure. Figure 7 A structural block diagram of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0021] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0022] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0023] To facilitate understanding, before introducing the embodiments of this disclosure, the following explanations are provided for several terms involved in the embodiments of this disclosure: SLAM: Simultaneous Localization and Mapping.

[0024] The specific implementation methods of the embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0025] Figure 1 This diagram illustrates a target tracking system architecture based on arbitrary binocular images, according to an embodiment of this disclosure. Figure 1 As shown, the system architecture may include a first vision device 10, a second vision device 20, and an image processing device 30.

[0026] The system comprises: a first vision device 10 for synchronously capturing images of the target object to obtain a first image; a second vision device 20 for capturing images of the target object to obtain a second image; and an image processing device 30 for acquiring the first and second images of the current frame obtained by the first and second vision devices simultaneously capturing images of the target object; performing image transposition processing on the first and second images to obtain a pseudo-horizontal binocular image, wherein the binocular baseline corresponding to the pseudo-horizontal binocular image is in the horizontal direction; inputting the pseudo-horizontal binocular image into the Simultaneous Localization and Mapping (SLAM) algorithm tracking module to output the current pose of the target object in the visual coordinate system; calculating the intermediate relative pose of the target object in the visual coordinate system based on the current pose and the keyframe pose; and performing axis-consistent processing on the intermediate relative pose to output the final relative pose in the control coordinate system.

[0027] In some embodiments, the image processing device 30 is further configured to: retrieve calibration intrinsic parameters and calibration extrinsic parameters, and perform image distortion correction and / or epipolar correction on the first image and the second image respectively.

[0028] In this embodiment, intrinsic parameters refer to the intrinsic parameters of the vision device, namely the inherent calibration parameters of the first or second vision device (including focal length, principal point coordinates, pixel ratio, and lens distortion correction parameters). Their core function is to accurately convert the pixel coordinates at the image level into coordinates in a three-dimensional vision coordinate system with the optical center of the vision device as the origin, while simultaneously eliminating coordinate deviations caused by lens distortion. Extrinsic parameters are calibration parameters (including translation and rotation matrices) that describe the relative pose relationships between different coordinate systems. This solution involves two layers of extrinsic parameters. The first layer is the extrinsic parameters between the first and second vision devices, such as the relative position and orientation of the binocular vision devices, ensuring that the anchor point coordinates after binocular image fusion are in a unified binocular vision coordinate system, avoiding coordinate misalignment caused by device installation deviations. The second layer is the extrinsic parameters between the binocular vision device and the boarding bridge control coordinate system: calibrating the translation and rotation relationships between the vision coordinate system of the calibration vision device and the control coordinate system of the boarding bridge (such as the motion coordinate system and docking reference coordinate system of the boarding bridge), directly determining whether the coordinates can be accurately mapped to the control space.

[0029] In some embodiments, the image processing device 30 is further configured to: obtain the angle between the binocular baselines of the first vision device 10 and the second vision device 20 and the horizontal plane; if the angle is 90°, then the first image and the second image are respectively matrix transposed so that the original pixel column direction is mapped to the pixel row direction and the original pixel row direction is mapped to the pixel column direction to obtain a pseudo-horizontal binocular image; if the angle is any other angle besides 0° and 90°, then the first image and the second image are respectively rotated and transposed according to the angle to obtain a pseudo-horizontal binocular image.

[0030] In some embodiments, the image processing device 30 is further configured to: perform downsampling processing on the pseudo-horizontal binocular image; and / or perform masking processing on the pseudo-horizontal binocular image to preserve the pixel region where the target object is located.

[0031] In some embodiments, when the target object is an aircraft door and the first vision device 10 and the second vision device 20 are binocular vision devices mounted on a boarding bridge, the image processing device 30 is further configured to: detect anchor points on the aircraft door frame in the keyframe image and obtain the keyframe anchor point coordinates; transform the keyframe anchor point coordinates to the current frame anchor point coordinate intermediate value through relative pose intermediate value; read calibration intrinsic parameters and calibration extrinsic parameters, perform axis-consistent processing on the current frame anchor point coordinate intermediate value, and output the final value of the current frame anchor point coordinates in the control coordinate system.

[0032] In some embodiments, the image processing device 30 is further configured to: read calibration intrinsic parameters and calibration extrinsic parameters, initialize the synchronous positioning and mapping (SLAM) algorithm tracking module, and disable loop closure detection.

[0033] In this embodiment of the disclosure, the medium providing the communication link between the first vision device 10, the second vision device 20, and the image processing device can be a wired network or a wireless network.

[0034] Optionally, the aforementioned wireless or wired networks use standard communication technologies and / or protocols. The network is typically the Internet, but can also be any network, including but not limited to Local Area Networks (LANs), Metropolitan Area Networks (MANs), Wide Area Networks (WANs), mobile, wired or wireless networks, private networks, or any combination of virtual private networks. In some embodiments, technologies and / or formats, including Hyper Text Markup Language (HTML), Extensible Markup Language (XML), etc., are used to represent data exchanged over the network. Furthermore, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Networks (VPNs), and Internet Protocol Security (IPSec) can be used to encrypt all or some links. In other embodiments, custom and / or dedicated data communication technologies can be used to replace or supplement the aforementioned data communication technologies.

[0035] In this embodiment of the disclosure, the image processing device 30 can be any type of electronic device, such as a terminal or a server.

[0036] In some embodiments, when the image processing device 30 is a terminal, the terminal may be a mobile phone, tablet computer, laptop computer, notebook computer, personal digital assistant (PDA), handheld computer, netbook, ultra-mobile personal computer (UMPC), mobile internet device (MID), augmented reality (AR), virtual reality (VR) device, robot, wearable device, flight vehicle, vehicle user equipment (VUE), shipboard equipment, pedestrian user equipment (PUE), smart home (home devices with wireless communication capabilities, such as refrigerators, televisions, washing machines, or furniture), game console, personal computer (PC), ATM, or self-service machine, etc. Wearable devices include: smartwatches, smart bracelets, smart earphones, smart glasses, smart jewelry (smart bracelets, smart necklaces, smart anklets, smart ankle chains, etc.), smart wristbands, smart clothing, etc. Among these, in-vehicle devices can also be referred to as in-vehicle terminals, in-vehicle controllers, in-vehicle modules, in-vehicle components, in-vehicle chips, or in-vehicle units, etc. It should be noted that the specific type of terminal is not limited in the embodiments disclosed herein.

[0037] In some embodiments, when the image processing device 30 is a server, the server may be a server that provides various services, such as a background management server that supports the device operated by the user using the terminal. The background management server can analyze and process received data such as requests, and feed the processing results back to the terminal device.

[0038] Optionally, the server can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0039] Those skilled in the art will know that Figure 1 The number of the first vision device, second vision device, and image processing device is merely illustrative; any number of these devices can be used depending on actual needs. This disclosure does not limit the scope of the embodiments.

[0040] Under the above system architecture, this disclosure provides a target tracking method based on arbitrary binocular images, which can be executed by any electronic device with computing power.

[0041] In some embodiments, the target tracking based on arbitrary binocular images provided in this disclosure can be performed by the terminal of the system architecture described above; in other embodiments, the target tracking based on arbitrary binocular images provided in this disclosure can be performed by the server in the system architecture described above; in still other embodiments, the target tracking based on arbitrary binocular images provided in this disclosure can be implemented by the terminal and the server in the system architecture described above through interaction.

[0042] Figure 2 This diagram illustrates a target tracking method based on arbitrary binocular images according to an embodiment of the present disclosure, such as... Figure 2 As shown, the target tracking method based on arbitrary binocular images provided in this embodiment includes the following steps: S202, acquire the first image and the second image of the current frame obtained by the first vision device and the second vision device simultaneously capturing images of the target object.

[0043] It should be noted that the first vision device and the second vision device in this embodiment are two imaging devices for binocular vision imaging, and can be any imaging device such as an industrial camera or a webcam. The target object can be any object being photographed, including but not limited to an aircraft door. In this embodiment, the first image and the second image are original images obtained simultaneously by the first vision device and the second vision device, respectively. Simultaneous shooting means that the first vision device and the second vision device acquire images at the same time, ensuring that the timestamps of the acquired first image and the second image are consistent (e.g., synchronization error ≤ 1ms). Simultaneous shooting can avoid feature misalignment caused by target object movement due to differences in acquisition timing, and ensure the accuracy of subsequent parallax calculation.

[0044] In one embodiment, when the target object is an aircraft door, the first vision device and the second vision device refer to binocular vision devices installed on the boarding bridge. In specific implementation, the first vision device and the second vision device are fixed to the front end of the boarding bridge in a parallel stacking manner, with the first vision device located on top and the second vision device located on the bottom. The two must ensure that their relative poses are fixed to form a stable vertical baseline.

[0045] S204, perform image transpose processing on the first image and the second image to obtain a pseudo-horizontal binocular image, wherein the binocular baseline corresponding to the pseudo-horizontal binocular image is in the horizontal direction.

[0046] In this embodiment, the pseudo-horizontal stereo image refers to an equivalent horizontal baseline image pair (left and right stereo images) obtained through transposition processing. Essentially, it involves transforming the row and column dimensions of the original vertical baseline image; it is not a physically horizontal baseline image, but it ensures that the projection points of the same feature point of the target object in both images are located in the same pixel row, satisfying the epipolar constraint requirements of the horizontal baseline SLAM algorithm. In this embodiment, image transposition processing refers to transposing any original stereo image to adapt it to the horizontal baseline SLAM algorithm. In one embodiment, for the top and bottom stereo images, the following processing can be performed to obtain the left and right stereo images: swapping pixel rows and columns to achieve an equivalent transformation from the vertical baseline to the horizontal baseline.

[0047] In this embodiment of the disclosure, the binocular baseline refers to the line connecting the centers of the first vision device and the second vision device. In the scenario of a boarding bridge docking with an aircraft door, the binocular images acquired by the upper and lower binocular vision devices originally have a vertical baseline (the line connecting the centers of the upper and lower cameras). After transposition processing, the binocular baseline of the pseudo-horizontal binocular image is equivalent to the horizontal baseline.

[0048] S206, input the pseudo-horizontal binocular image into the Simultaneous Localization and Mapping (SLAM) algorithm tracking module, and output the current pose of the target object in the visual coordinate system.

[0049] It should be noted that the SLAM algorithm tracking module can calculate the device's own position and construct a map of the surrounding environment in real time based on the input visual data (such as parallax and feature points). In this embodiment, an improved ORB-SLAM2 tracking module adapted to the vertical baseline is used, loop closure detection is turned off and motion prior is optimized, focusing on the real-time tracking and positioning of the door. Taking the docking of the boarding bridge with the aircraft door as an example, the spatial coordinate system refers to the coordinate system adapted to the docking requirements of the boarding bridge, usually the coordinate system of the boarding bridge robot arm or the coordinate system of the door (preset coordinate system). Its coordinate axis direction is defined according to engineering requirements (such as the X-axis along the forward direction and the Z-axis perpendicular to the ground), used to accurately describe the spatial position of the target object. The position information of the target object (such as the aircraft door) may include, but is not limited to, the three-dimensional coordinate information and attitude information (rotation matrix R, translation vector t) of the target object (such as the aircraft door) in the spatial coordinate system, which is the core data for the boarding bridge to perform the docking action.

[0050] S208, calculate the relative pose intermediate of the target object in the visual coordinate system based on the current pose and the keyframe pose in the visual coordinate system.

[0051] It should be noted that the current pose refers to the absolute pose of the target object (aircraft door) in the visual coordinate system at the moment of the current frame capture, as output by the SLAM algorithm tracking module (used to describe the real-time position and attitude of the aircraft door in visual space); the keyframe pose refers to the pre-selected tracking reference, that is, the absolute pose of the target object in the visual coordinate system at the moment of the keyframe capture (serving as the reference for pose comparison, and all pose changes in subsequent frames are referenced to this pose); the relative pose intermediate quantity refers to the relative change in pose of the target object from the keyframe to the current frame, obtained by calculating the pose difference between the current pose and the keyframe pose, and is also a parameter in the visual coordinate system. The relative pose intermediate quantity is only used to describe the motion state (position change or rotation change) of the target object from the keyframe to the current frame, and is not adapted to the coordinate system requirements of industrial control, and cannot be directly used for boarding bridge control.

[0052] S210 performs axis-consistent processing on the intermediate relative pose values ​​to output the final relative pose values ​​in the control coordinate system.

[0053] This step is the core transformation step from the vision algorithm layer to the industrial control layer. It achieves pose mapping from the vision coordinate system to the control coordinate system through axis consistency, transforming intermediate relative pose quantities that cannot be directly used into final relative pose quantities that the boarding bridge control system can directly call. The vision coordinate system refers to the default output coordinate system of the SLAM algorithm, a spatial coordinate system with the optical center of the boarding bridge's binocular vision device as the origin; the control coordinate system is the industrial motion coordinate system of the boarding bridge (with a freely selected reference point on the boarding bridge, such as the boarding bridge docking reference point / motion origin, as the origin, and the coordinate axes correspond one-to-one with the boarding bridge's forward / backward / left / right / up / down motion axes and rotation axes, such as the X-axis for the boarding bridge's extension and retraction direction, the Y-axis for the left and right adjustment direction, and the Z-axis for the up and down adjustment direction).

[0054] In this embodiment of the disclosure, axis unification processing refers to rotating and matching the coordinate axis directions of the visual coordinate system to the coordinate axis directions of the control coordinate system, and simultaneously completing the spatial position mapping of the two coordinate systems (including rotation matrix transformation and translation vector compensation).

[0055] In this embodiment, axis consistency processing eliminates the axial direction and origin deviations between the visual coordinate system (centered on the camera) and the control coordinate system (centered on the boarding bridge), ensuring that the physical meaning of the pose data aligns with the motion logic of the boarding bridge (e.g., the X-axis of the visual coordinate system, which is the camera's optical axis, is converted to the X-axis of the control coordinate system, which is the boarding bridge's extension / retraction direction). Transforming the "relative pose intermediate quantity" at the visual algorithm level into the "relative pose final quantity" at the industrial control level is the final crucial step connecting visual tracking and the automatic docking of the boarding bridge. In practical implementation, axis consistency and coordinate system transformation can be performed based on pre-calibrated extrinsic parameters of the binocular vision device and the boarding bridge, eliminating errors caused by equipment installation and coordinate system deviations, and ensuring that the output final pose data meets the millimeter-level accuracy requirements for boarding bridge docking.

[0056] In one embodiment, the extrinsic parameters of the binocular vision device and the boarding bridge include a rotation matrix R from the visual coordinate system to the control coordinate system and a translation vector T. Using these extrinsic parameters as the basis for transformation, the intermediate relative pose quantity in the visual coordinate system is made axially consistent: the coordinate axis direction of the intermediate quantity is rotated by the rotation matrix R to be completely consistent with the motion axis of the control coordinate system, eliminating the axial direction deviation between the visual coordinate system and the control coordinate system; the spatial position compensation of the intermediate quantity is performed by the translation vector T to complete the position mapping from the visual coordinate system to the control coordinate system; after the above processing, the final relative pose quantity in the control coordinate system is output, whose parameter dimensions and axial directions are completely matched with the motion control logic of the boarding bridge.

[0057] In some embodiments, before performing image transpose processing on the first image and the second image, the target tracking method based on arbitrary binocular images provided in this disclosure may further include the following steps: retrieving calibration intrinsic parameters and calibration extrinsic parameters, and performing image distortion correction and / or epipolar correction on the first image and the second image respectively. Here, the intrinsic parameters are the camera intrinsic parameters (such as distortion coefficients) of the first vision device and the second vision device; the extrinsic parameters are the camera extrinsic parameters (such as the relative pose between the binocular vision devices: relative rotation matrix and translation vector).

[0058] To address the hardware distortion and binocular imaging misalignment issues present in the original first and second images captured by binocular vision devices, image distortion correction and epipolar correction operations are added before image transposition processing. Accurate calibration of the original images is achieved using camera intrinsic and / or extrinsic parameters, ensuring a qualified pseudo-horizontal binocular image for subsequent image transposition processing. This guarantees the tracking accuracy of the SLAM algorithm and improves the robustness and accuracy of the entire tracking method. Typically, intrinsic parameters are used to address image errors caused by the device's own hardware, while extrinsic parameters are used to address relative misalignment issues arising from the installation / imaging of binocular vision devices. The two can be combined as needed to adapt to device scenarios with different calibration accuracies.

[0059] Due to manufacturing and processing errors, camera lenses can produce images with radial distortion (such as barrel or pincushion distortion) and tangential distortion (such as image tilt), causing the target contours and feature point positions in the image to deviate from their actual physical positions. In this embodiment, by retrieving camera intrinsic parameters (including but not limited to distortion correction coefficients, focal length, principal point coordinates, etc.) and utilizing the distortion correction model corresponding to these intrinsic parameters, pixel-level geometric correction is performed on the first and second images respectively. This eliminates image distortion caused by lens distortion, restores the true visual features of the target object (such as an aircraft door), and ensures the imaging accuracy of a single image.

[0060] Due to installation deviations (such as non-parallel optical axes or non-coplanar imaging planes), the first and second images captured by the two cameras of a binocular vision device may exhibit issues such as non-parallel epipolar lines and a large search range for corresponding feature points. Directly using these images for subsequent binocular matching and SLAM tracking would significantly increase the computational load and reduce matching accuracy. In this embodiment, by retrieving camera extrinsic parameters (including but not limited to the rotation matrix and translation vector of the binocular vision device), a coordinated geometric transformation is performed on the first and second images. This makes the epipolar lines of the binocular images more horizontal and parallel, ensuring that the row coordinates of the same target feature point remain consistent across the two images. This achieves binocular image alignment, providing a precise coordinated image foundation for the subsequent generation of pseudo-horizontal binocular images and binocular feature matching in the SLAM algorithm.

[0061] After performing image distortion correction and epipolar correction, the calibrated, distortion-free, and epipolar-aligned original image will be used as input data for subsequent image transposition processing. This can avoid deviations in the transposed pseudo-horizontal binocular image caused by distortion / misalignment of the original image, which would affect the accuracy of subsequent SLAM pose tracking.

[0062] In this embodiment, the original binocular image undergoes distortion correction and / or epipolar correction preprocessing before image transposition. Camera intrinsic parameters eliminate image distortion caused by lens hardware, restoring the true visual features of the target object. Camera extrinsic parameters achieve epipolar alignment of the binocular image, resolving binocular imaging misalignment issues. This not only improves the imaging accuracy of the original binocular image but also adapts to binocular vision devices with different calibration accuracies, enhancing the method's device compatibility. Simultaneously, the high-quality preprocessed image provides a precise foundation for subsequent pseudo-horizontal binocular image generation, SLAM algorithm feature matching, and pose tracking. This effectively reduces algorithm computational complexity and the probability of feature point mismatch, minimizing the accumulation of errors in subsequent pose calculations and coordinate transformations. From the source, this improves the positioning accuracy, computational efficiency, and robustness in complex scenarios of the entire target tracking method, providing crucial image assurance for the final output of accurate pose / coordinate data that meets industrial control requirements.

[0063] It's important to note that if only intrinsic parameters are retrieved, only distortion correction is performed on the image, suitable for scenarios where extrinsic parameters are accurately calibrated and binocular imaging is basically aligned. If only extrinsic parameters are retrieved, only epipolar correction is performed on the image, suitable for high-precision equipment scenarios with minimal or negligible lens distortion. If both intrinsic and extrinsic parameters are retrieved simultaneously, distortion correction and epipolar correction can be performed concurrently, suitable for conventional industrial equipment scenarios, achieving the highest precision image calibration. Pre-calibrating the original binocular image using both intrinsic and extrinsic parameters—achieving "monocular vanishing truth and binocular alignment"—provides a high-quality image foundation for all subsequent core processing steps.

[0064] In some embodiments, such as Figure 3 As shown, the target tracking method based on arbitrary binocular images provided in this embodiment can perform image transpose processing on the first image and the second image through the following steps to obtain a pseudo-horizontal binocular image: S302, Obtain the angle between the binocular baselines of the first vision device and the second vision device and the horizontal plane; S304, if the included angle is 90°, then the first image and the second image are matrix transposed respectively, so that the original pixel column direction is mapped to the pixel row direction and the original pixel row direction is mapped to the pixel column direction, thus obtaining a pseudo-horizontal binocular image; S306, if the included angle is any other than 0° and 90°, then the first image and the second image are rotated and transposed according to the included angle to obtain a pseudo-horizontal binocular image.

[0065] In the above embodiments, the angle between the baseline and the horizontal direction is used to accurately distinguish three baseline types: horizontal (angle of 0°), vertical (angle of 90°), and tilted (angles other than 0° and 90°). Differential correction strategies are designed accordingly: the horizontal baseline is not transposed; the vertical baseline achieves parallax direction conversion through matrix transposition; and the tilted baseline ensures epipolar constraint through epipolar alignment. This design of three baseline types can adapt to the installation requirements of binocular devices in different scenarios such as boarding airports, mobile robots, and drones. Whether in a vertically stacked layout with limited installation space or a tilted layout in a complex environment, it can stably output pseudo-horizontal binocular images, significantly expanding the hardware adaptability of the technical solution.

[0066] In this embodiment, a transposition process is used to uniformly convert stereo images of any baseline type into pseudo-horizontal stereo images. This design allows the SLAM algorithm relied upon for subsequent target tracking to directly reuse a mature horizontal baseline SLAM kernel, without needing to reconstruct the core logic of the algorithm for different baseline types. Compared to the traditional approach that requires developing separate adaptation algorithms for different baselines, this embodiment significantly reduces the algorithm development cycle and maintenance costs. Simultaneously, the reuse of a mature SLAM kernel ensures the stability of target tracking, avoids the robustness risks associated with developing new algorithms, and accelerates the engineering implementation of the technical solution.

[0067] In some embodiments, before inputting the pseudo-horizontal stereo image into the SLAM algorithm tracking module, the method further includes: downsampling the pseudo-horizontal stereo image; and / or masking the pseudo-horizontal stereo image to preserve the pixel region where the target object is located.

[0068] It should be noted that downsampling can reduce the resolution of the pseudo-horizontal binocular image (e.g., from 1920×1080 to 960×540), significantly reducing the computational load of feature extraction and matching without affecting the core features of the target. Actual measurements show that feature extraction time is reduced by approximately 40% after downsampling. Combined with a lightweight correction process, the entire visual front-end (correction + disparity calculation) can be kept within 5ms, fully meeting the millisecond-level real-time requirements for tasks such as boarding and disembarking aircraft doors. After performing downsampling on the masked pseudo-horizontal binocular image, this embodiment calculates the equivalent horizontal disparity based on accurate projection point coordinates, avoiding interference from irrelevant feature points in the calculation results.

[0069] Masking can remove interference and improve the purity of feature matching. In this embodiment, masking accurately preserves the pixel area where the target object (such as an aircraft door) is located, filters out irrelevant pixels such as the sky, fuselage, and background clutter, reduces noise interference during subsequent feature extraction, makes the extracted target feature points more focused, reduces the probability of mismatch from the source, and ensures the effectiveness of parallax calculation.

[0070] In some embodiments, the target object is the aircraft door, and the first and second vision devices are binocular vision devices mounted on the boarding bridge, such as... Figure 4 As shown, the target tracking method based on arbitrary binocular images provided in this embodiment may further include the following steps: S402, detect anchor points on the aircraft door frame in the keyframe image and obtain the keyframe anchor point coordinates; S404, transform the keyframe anchor point coordinates to the current frame anchor point coordinates using the relative pose intermediate value; S406 reads the calibration intrinsic parameters and calibration extrinsic parameters, performs axis consistency processing on the intermediate values ​​of the current frame anchor point coordinates, and outputs the final values ​​of the current frame anchor point coordinates in the control coordinate system.

[0071] It should be noted that, in this embodiment, the anchor point is a feature point selected on the aircraft door for docking. This feature point is typically a feature point with a fixed geometric position and high visual recognition (such as the left and right corners of the bottom of the door frame, the midpoint of the edge, or the welding positioning point). It should also be noted that, in this embodiment, the step of detecting the anchor points on the aircraft door frame in the keyframe image and obtaining the keyframe anchor point coordinates can be performed simultaneously with pose calculation, or before or after pose calculation. This embodiment does not specifically limit the execution order of this step.

[0072] In this embodiment, axis alignment refers to rotating the camera's coordinate axes to be parallel to the robot's coordinate axes using a rotation matrix of the calibration extrinsic parameters, and translating the origin of the coordinate system to the robot's origin using a translation vector. The calibration intrinsic parameters in S406 above refer to the inherent parameters of the first and second vision devices (such as distortion coefficients, used to eliminate image distortion); the calibration extrinsic parameters refer to the rigid transformation parameters between the camera coordinate system and the control coordinate system, including at least the rotation matrix and translation vector. It should be noted that the calibration extrinsic parameters here are not the relative pose between the first and second vision devices, but rather the pose of the vision device relative to the actuator (such as a boarding bridge), used to convert the anchor point coordinates in the camera coordinate system to coordinates in the control coordinate system through rotation and translation transformations.

[0073] In some embodiments, such as Figure 5As shown in the embodiments of this disclosure, the target tracking method based on arbitrary binocular images may further include the following steps: S502, read calibration intrinsic parameters and calibration extrinsic parameters, initialize the synchronous positioning and mapping SLAM algorithm tracking module and disable loop closure detection.

[0074] This embodiment initializes the SLAM algorithm tracking module by reading calibration intrinsic and extrinsic parameters, ensuring precise matching between the algorithm's computation and the optical and spatial calibration parameters of the vision device. This eliminates system errors caused by mismatched device parameters from the startup phase, guaranteeing initial accuracy and operational consistency in pose tracking. Simultaneously, to address the short-distance, high-real-time tracking requirements of boarding bridge docking, loop closure detection is disabled, omitting redundant loop closure verification computations. This significantly improves the pose calculation speed and real-time output of the SLAM module, enabling rapid response to changes in aircraft door pose. Furthermore, the simplified algorithm computation logic reduces pose jump problems caused by loop closure mismatches, improving the module's tracking stability in complex airport environments. It also forms a closed loop with the overall technical solution's device calibration and image preprocessing steps, ensuring end-to-end accuracy and efficiency from image input to control data output, thus meeting the industrial control requirements of automatic boarding bridge docking.

[0075] The core function of loop closure detection is to eliminate the accumulated error of long-distance tracking. However, it requires extensive feature matching and pose verification between historical frames and the current frame, which significantly increases the computation time of the algorithm. In contrast, the docking of the boarding bridge and the aircraft door is a short-distance, high-real-time dynamic tracking scenario, which does not require long-distance loop closure correction. By disabling this function, the SLAM module can omit redundant computations and focus on pose tracking between the current frame and key frames, significantly improving the algorithm's computation speed and the real-time performance of pose output. This ensures rapid response to minor pose changes in the aircraft door and meets the real-time control requirements of automatic boarding bridge docking. Disabling loop closure detection simplifies the SLAM module's computational logic, reducing pose jumps and tracking drift caused by mismatches in loop closure detection (such as feature misjudgments due to similar airport scenes), making the module more stable in the complex visual environment of an airport. Simultaneously, combined with precise initialization of the device's intrinsic and extrinsic parameters, it further reduces invalid computations and improves the continuity and reliability of the module's pose tracking of the target object (aircraft door).

[0076] Since the pseudo-horizontal binocular image input to the SLAM algorithm tracking module has undergone optimization processing such as masking and downsampling, it provides reliable 3D data support for subsequent pose calculation. In this embodiment, the pseudo-horizontal binocular image is input into the SLAM algorithm tracking module. Compared with the lack of scale information in traditional monocular SLAM, this combined data can provide clear 3D spatial constraints, effectively suppressing the drift of SLAM pose calculation. At the same time, combined with the characteristics of the pseudo-horizontal image after correction, the SLAM module can reuse mature horizontal baseline feature matching and optimization logic to further improve the stability of pose calculation. Even under complex conditions of alternating rotation and forward movement during boarding, it can stably output the pose information (rotation matrix, translation vector) of the target object in the visual coordinate system.

[0077] Furthermore, in this embodiment, the pose information in the visual coordinate system is mapped to a spatial coordinate system (such as the coordinate system of the boarding robot arm or the preset coordinate system of the hatch) through coordinate system transformation, which solves the problem of inconsistency between the visual coordinate system and the engineering execution coordinate system. The transformation process can achieve accurate mapping based on pre-calibrated equipment extrinsic parameters without additional complex calculations, so that the output position information can directly guide the execution mechanisms such as the boarding robot arm docking and the robot target grasping.

[0078] Taking the automatic docking scenario of airport boarding bridge and aircraft door as an example, the target tracking method based on arbitrary binocular images provided in this embodiment can solve the adaptation problem of traditional horizontal baseline SLAM algorithm in scenarios with limited installation space and special motion modes. Through vertical baseline stereo vision design, loop closure detection and removal, and velocity prior adaptive adjustment, high-precision and robust positioning and docking guidance of the door can be achieved.

[0079] The traditional ORB-SLAM2 algorithm is the mainstream visual SLAM solution for general scenarios. Its core design is based on a "horizontal baseline (left and right cameras)" layout, relies on loop closure detection to suppress pose drift, and defaults to using a uniform motion model as a prior constraint for feature matching between adjacent frames. However, in the boarding and docking scenario, this solution has significant adaptation defects, specifically manifested as follows: 1) Camera layout incompatibility: The installation space at the front end of the boarding and receiving robotic arm is narrow, and the binocular camera cannot be arranged horizontally. It can only adopt a vertical baseline layout with the cameras stacked one on top of the other, which causes the epipolar geometry to change from "near horizontal" to "near vertical". Directly using the horizontal baseline algorithm will result in feature matching direction and correction model mismatch problems. 2) Imbalance between loop closure cost and performance: The motion trajectory of boarding and receiving aircraft relative to the cabin door is a non-closed trajectory with "monotonically decreasing distance". The drift suppression benefit of loop closure detection is extremely low, but it will consume a lot of computing resources and affect the real-time performance of embedded devices. 3) Motion prior conflict: The boarding and docking control strategy is an alternating iterative mode of "rotation → forward → rotation → forward". At the point of motion mode switching, the uniform motion prior is completely invalidated, which can easily introduce matching outside points and induce tracking failure.

[0080] Based on this, the embodiments of this disclosure specifically design a vertical baseline adaptation pipeline and corresponding optimization strategies to address the aforementioned technical pain points. The target tracking method based on arbitrary binocular images provided in the embodiments of this disclosure can achieve, but is not limited to, the following technical effects: 1) SLAM algorithm tracking adapted to vertical baseline binocular images improves the positioning accuracy of aircraft doors in boarding and docking scenarios; 2) While eliminating loop closure detection, reduce system memory usage and average computation latency to ensure the continuity of SLAM tracking; 3) For the rotation-forward alternating control strategy, the matching error at the mode switching point is reduced by disabling / adaptively weakening the speed prior, thereby improving the overall tracking robustness; 4) Integrate the geometric anchor point information of the hatch to improve the positioning stability and accuracy of the docking end.

[0081] In practical implementation, the first vision device 10 is installed above the industrial camera at the front end of the boarding bridge robotic arm. The second vision device 20 is a parallel stack of the same model industrial camera as the first vision device 10, with a baseline length B = 50mm (vertical distance between the optical centers of the two cameras). The industrial camera used can be a Basler acA1920-40gc with a resolution of 1920×1080, a frame rate of 30fps, a lens focal length of 8mm, and strong light interference resistance. The image processing device 30 uses an embedded industrial computer (CPU: Intel Core i7-12700H, GPU: NVIDIA RTX 3050, 16GB memory) to run SLAM algorithms, image preprocessing, and pose calculation. The image processing device 30 can interface with the boarding bridge control system to input the door pose information to the boarding bridge control system, which is used to further drive the boarding bridge to move closer to the aircraft for boarding.

[0082] In one embodiment, the calibration parameters of the upper and lower binocular cameras include, but are not limited to, any one of the following: ① Intrinsic parameter matrix K (consistent for both cameras); ② Distortion coefficient; ③ Relative rotation matrix R (no additional rotation for vertical stacking); ④ Relative translation vector T (along the vertical direction, consistent with the baseline length).

[0083] The target tracking method based on arbitrary binocular images provided in this disclosure, when applied to an automatic docking scenario between airport boarding bridges and aircraft doors, may specifically include the following steps: 1) Vertical baseline dual-target localization and image preprocessing: 1.1) Start the first vision device 10 and the second vision device 20 to synchronously acquire the original image pair (first image, second image) of the target object (aircraft door) and ensure that the timestamps of the two images are consistent (synchronization error ≤ 1ms). 1.2) Based on the distortion coefficients in the calibration parameters, perform distortion removal processing on the first image and the second image respectively. In specific implementation, the cv2.undistort function of OpenCV can be used to eliminate lens optical distortion and obtain the upper and lower images after distortion removal. 1.3) Perform pixel matrix transpose operation on the distortion-free images: interchange the row and column dimensions of the first and second images to obtain a pseudo-horizontal binocular image pair (left and right images), realizing the conversion of vertical row coordinate disparity to horizontal column coordinate disparity.

[0084] 2) Pseudo-horizontal image enhancement and feature extraction: 2.1) Perform masking on the pseudo-horizontal binocular image pair (left and right images): Based on the prior dimensions of the hatch (e.g., width 1.8m, height 2.2m), generate a binary mask, retain only the pixel area where the hatch is located, and filter out background interference pixels such as the fuselage and sky; 2.2) Perform downsampling on the masked image: downsample the image resolution from 1920×1080 to 960×540 to reduce the computational load of subsequent feature extraction; 2.3) The ORB algorithm is used to extract feature points (key points + descriptors) from the left and right images after downsampling. The maximum number of feature points is set to 2000 and the number of pyramid layers is 8.

[0085] 3) Vertical baseline parallax and depth calculation: 3.1) Match the ORB feature points of the left and right images using a brute-force matcher (BFMatcher) and remove mismatched outliers using the RANSAC algorithm (e.g., 1000 iterations and a reprojection error threshold of 2.0 pixels). 3.2) Calculate the equivalent level disparity: The equivalent level disparity can be calculated using the following formula (1); 3.3) Depth calculation: The depth of the feature point in the camera coordinate system can be calculated by the above formula (2), f=1200 (calibration intrinsic parameter), B=50mm (baseline length); for the extremely small disparity segment with d<5 pixels, the Huber robust loss function is used to optimize the depth calculation results to avoid noise interference.

[0086] (1) (2) in, Represents the equivalent horizontal disparity of a pseudo-horizontal binocular image; This represents the pixel row coordinates of the projection point of the target feature point in the first image; The pixel row coordinates of the projection points of the target feature points in the corrected second image; This indicates the depth of the target feature points in the visual device's coordinate system; Indicates the equivalent focal length; This indicates the baseline length between the first vision device and the second vision device.

[0087] The method described in this disclosure was tested in a real-world boarding and arrival cabin door docking scenario, and the following beneficial effects were verified: ① Engineering reusability: Vertical baseline images can be adapted to the traditional horizontal baseline ORB-SLAM2 kernel simply by image transposition, without the need for reconstruction algorithms, reducing the adaptation cycle by 80%; ② Docking accuracy: After the door frame anchor points are integrated, the center positioning error of the hatch is ≤3mm, which meets the docking accuracy requirement of ±5mm for boarding and receiving aircraft; ③ Portability: By adjusting the calibration parameters and mask size, this solution can be quickly adapted to the door docking of different aircraft models such as A320 and B737, with an adaptation time of ≤24 hours.

[0088] Compared with the conventional ORB-SLAM2, the embodiments disclosed herein have, but are not limited to, the following advantages: 1) Stereo geometry adaptation: Breaking through the limitations of horizontal baselines, it adopts a vertical baseline layout and achieves pseudo-horizontal parallax through image transposition. It seamlessly reuses the existing SLAM kernel in engineering, which is different from the horizontal baseline dependence of traditional solutions. 4) Scenario-based precision optimization: Introducing soft constraint fusion of the geometric anchor points of the hatch frame, and customizing optimization for docking scenarios to improve the stability of end-point positioning, which is different from the undifferentiated optimization of general scenarios.

[0089] It should be noted that the acquisition, storage, use, and processing of data in this disclosed technical solution comply with the relevant provisions of laws and regulations. All types of data, such as personal identity data, operational data, and behavioral data related to individuals, customers, and groups, obtained in this disclosed embodiment have been agreed upon by the users.

[0090] Based on the same inventive concept, this disclosure also provides a target tracking device based on arbitrary binocular images, as described in the following embodiments. Since the principle by which this device solves the problem is similar to that of the method embodiments described above, the implementation of this device embodiment can refer to the implementation of the method embodiments described above, and repeated details will not be elaborated further.

[0091] Figure 6 This illustration shows a schematic diagram of a target tracking device based on arbitrary binocular images according to an embodiment of the present disclosure, such as... Figure 6 As shown, the device includes: an image acquisition module 601, an image transpose processing module 602, a SLAM tracking module 603, a relative pose intermediate quantity determination module 604, and a relative pose final quantity determination module 605.

[0092] The system includes: an image acquisition module 601, used to acquire the first and second images of the current frame obtained by the first and second vision devices simultaneously capturing images of the target object; an image transpose processing module 602, used to perform image transpose processing on the first and second images to obtain a pseudo-horizontal binocular image, wherein the binocular baseline corresponding to the pseudo-horizontal binocular image is in the horizontal direction; a SLAM tracking module 603, used to input the pseudo-horizontal binocular image to the Simultaneous Localization and Mapping (SLAM) algorithm tracking module and output the current pose of the target object in the visual coordinate system; a relative pose intermediate quantity determination module 604, used to calculate the relative pose intermediate quantity of the target object in the visual coordinate system based on the current pose and the keyframe pose; and a relative pose final quantity determination module 605, used to perform axis-consistent processing on the relative pose intermediate quantity to output the relative pose final quantity in the control coordinate system.

[0093] In some embodiments, the target tracking device based on arbitrary binocular images provided in this disclosure further includes: an image correction module 606, used to: retrieve calibration intrinsic parameters and calibration extrinsic parameters, and perform image distortion correction and / or epipolar correction on the first image and the second image respectively.

[0094] In some embodiments, the image transpose processing module 602 is further configured to: obtain the angle between the binocular baselines of the first vision device and the second vision device and the horizontal plane; if the angle is 90°, perform matrix transpose on the first image and the second image respectively, so that the original pixel column direction is mapped to the pixel row direction and the original pixel row direction is mapped to the pixel column direction, to obtain a pseudo-horizontal binocular image; if the angle is any other angle besides 0° and 90°, perform rotation transpose on the first image and the second image respectively according to the angle to obtain a pseudo-horizontal binocular image.

[0095] In some embodiments, the target tracking device based on arbitrary binocular images provided in this disclosure further includes: a pseudo-horizontal binocular image processing module 607, used for: performing downsampling processing on the pseudo-horizontal binocular image; and / or performing masking processing on the pseudo-horizontal binocular image to retain the pixel region where the target object is located.

[0096] In some embodiments, the target object is an aircraft door, and the first vision device and the second vision device are binocular vision devices mounted on a boarding bridge. The target tracking device based on arbitrary binocular images provided in this embodiment further includes: an anchor point coordinate determination module 608, used to detect anchor points on the aircraft door frame in the key frame image and obtain the key frame anchor point coordinates; transform the key frame anchor point coordinates to the current frame anchor point coordinate intermediate value through relative pose intermediate value; read calibration intrinsic parameters and calibration extrinsic parameters, and perform axis consistency processing on the current frame anchor point coordinate intermediate value to output the final value of the current frame anchor point coordinates in the control coordinate system.

[0097] In some embodiments, the target tracking device based on arbitrary binocular images provided in this disclosure further includes: a SLAM algorithm improvement module 609, used to read the camera intrinsic parameters and camera extrinsic parameters of the first vision device and the second vision device, initialize the synchronous localization and mapping SLAM algorithm tracking module, and disable loop closure detection.

[0098] In some embodiments, the first vision device and the second vision device are binocular vision devices, with the first vision device located above and the second vision device located below. The SLAM tracking module 603 is also used to calculate the depth of the target feature point in the visual coordinate system using the above formulas (1) and (2).

[0099] It should be noted that the examples and application scenarios implemented by the modules in the above device embodiments and the corresponding steps in the method embodiments are the same, but are not limited to the content disclosed in the above method embodiments. It should also be noted that the above modules, as part of the device, can be executed in a computer system such as a set of computer-executable instructions.

[0100] Those skilled in the art will understand that various aspects of this disclosure can be implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which can be collectively referred to herein as a "circuit", "module" or "system".

[0101] Based on the same inventive concept, this disclosure also provides an electronic device, which includes: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any of the above-described target tracking methods based on arbitrary binocular images by executing the executable instructions. Since the principle by which this electronic device solves the problem is similar to that of the above-described method embodiments, the implementation of this electronic device embodiment can refer to the implementation of the above-described method embodiments, and repeated details will not be elaborated further.

[0102] The following reference Figure 7 To describe an electronic device 700 according to such an embodiment of the present disclosure. Figure 7 The electronic device 700 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0103] like Figure 7 As shown, the electronic device 700 is manifested in the form of a general-purpose computing device. The components of the electronic device 700 may include, but are not limited to: at least one processing unit 710, at least one storage unit 720, and a bus 730 connecting different system components (including storage unit 720 and processing unit 710).

[0104] The storage unit stores program code that can be executed by the processing unit 710, causing the processing unit 710 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure. For example, the processing unit 710 can perform the following steps of the above method embodiments: acquiring a first image and a second image of the current frame obtained by a first vision device and a second vision device simultaneously capturing images of a target object; performing image transposition processing on the first image and the second image to obtain a pseudo-horizontal binocular image, wherein the binocular baseline corresponding to the pseudo-horizontal binocular image is in the horizontal direction; inputting the pseudo-horizontal binocular image to the Simultaneous Localization and Mapping (SLAM) algorithm tracking module, and outputting the current pose of the target object in the visual coordinate system; calculating the relative pose intermediate value of the target object in the visual coordinate system based on the current pose and the keyframe pose; performing axis-consistent processing on the relative pose intermediate value to output the final relative pose value in the control coordinate system.

[0105] Storage unit 720 may include readable media in the form of volatile storage units, such as random access memory (RAM) 7201 and / or cache memory 7202, and may further include read-only memory (ROM) 7203.

[0106] The storage unit 720 may also include a program / utility 7204 having a set (at least one) program module 7205, such program module 7205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0107] Bus 730 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0108] Electronic device 700 can also communicate with one or more external devices 740 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 700, and / or with any device that enables electronic device 700 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 750. Furthermore, electronic device 700 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 760. As shown, network adapter 760 communicates with other modules of electronic device 700 via bus 730. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 700, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0109] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0110] Based on the same inventive concept, this disclosure also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the target tracking method based on any of the above-described methods. Since the principle by which this computer-readable storage medium solves the problem is similar to that of the above-described method embodiments, the implementation of this computer-readable storage medium embodiment can refer to the implementation of the above-described method embodiments, and repeated details will not be elaborated further.

[0111] More specific examples of computer-readable storage media in this disclosure may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0112] In this disclosure, a computer-readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of transmitting, propagating, or transmitting a program for use by or in connection with an instruction execution system, apparatus, or device.

[0113] Optionally, the program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0114] In practical implementation, program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0115] Based on the same inventive concept, this disclosure also provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements any one of the target tracking methods based on arbitrary binocular images in the above method embodiments. Since the principle by which this computer program product embodiment solves the problem is similar to that of the above method embodiments, the implementation of this computer program product embodiment can refer to the implementation of the above method embodiments, and repeated details will not be elaborated further.

[0116] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0117] Furthermore, although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0118] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0119] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.

Claims

1. A target tracking method based on arbitrary binocular images, characterized in that, include: Acquire the first and second images of the current frame obtained by the first and second vision devices simultaneously capturing images of the target object; The first image and the second image are transposed to obtain a pseudo-horizontal binocular image, wherein the binocular baseline corresponding to the pseudo-horizontal binocular image is in the horizontal direction; The pseudo-horizontal binocular image is input into the Simultaneous Localization and Mapping (SLAM) algorithm tracking module, which outputs the current pose of the target object in the visual coordinate system. Calculate the relative pose intermediate value of the target object in the visual coordinate system based on the current pose and the keyframe pose in the visual coordinate system; The intermediate relative pose values ​​are processed to achieve axis consistency, so as to output the final relative pose values ​​in the control coordinate system.

2. The target tracking method based on arbitrary binocular images according to claim 1, characterized in that, Before performing image transpose processing on the first image and the second image, the method further includes: Retrieve calibration intrinsic parameters and calibration extrinsic parameters, and perform image distortion correction and / or epipolar correction on the first image and the second image respectively.

3. The target tracking method based on arbitrary binocular images according to claim 1, characterized in that, The step of performing image transpose processing on the first image and the second image to obtain a pseudo-horizontal binocular image includes: Obtain the angle between the binocular baselines of the first vision device and the second vision device relative to the horizontal plane; If the included angle is 90°, then the first image and the second image are respectively matrix transposed so that the original pixel column direction is mapped to the pixel row direction and the original pixel row direction is mapped to the pixel column direction, thus obtaining a pseudo-horizontal binocular image; If the included angle is any angle other than 0° and 90°, then the first image and the second image are rotated and transposed according to the included angle to obtain a pseudo-horizontal binocular image.

4. The target tracking method based on arbitrary binocular images according to claim 1, characterized in that, Before inputting the pseudo-horizontal stereo image into the Simultaneous Localization and Mapping (SLAM) algorithm tracking module, the method further includes: The pseudo-horizontal binocular image is downsampled. And / or, perform masking processing on the pseudo-horizontal binocular image to preserve the pixel region where the target object is located.

5. The target tracking method based on arbitrary binocular images according to claim 1, characterized in that, The target object is an aircraft cabin door, and the first and second vision devices are binocular vision devices mounted on a boarding bridge. The method further includes: Detect anchor points on the aircraft door frame in the keyframe image and obtain the keyframe anchor point coordinates; The keyframe anchor point coordinates are transformed to the current frame anchor point coordinate intermediate value using the relative pose intermediate value; Read the calibration intrinsic parameters and calibration extrinsic parameters, perform axis consistency processing on the intermediate values ​​of the current frame anchor point coordinates, and output the final values ​​of the current frame anchor point coordinates in the control coordinate system.

6. The target tracking method based on arbitrary binocular images according to claim 1, characterized in that, The method further includes: Read the calibration intrinsic parameters and calibration extrinsic parameters, initialize the synchronous localization and map building SLAM algorithm tracking module and disable loop closure detection.

7. A target tracking device based on arbitrary binocular images, characterized in that, include: The image acquisition module is used to acquire the first image and the second image of the current frame obtained by the first vision device and the second vision device simultaneously capturing images of the target object; The image transpose processing module is used to perform image transpose processing on the first image and the second image to obtain a pseudo-horizontal binocular image, wherein the binocular baseline corresponding to the pseudo-horizontal binocular image is in the horizontal direction; The SLAM tracking module is used to input the pseudo-horizontal binocular image into the Simultaneous Localization and Mapping (SLAM) algorithm tracking module and output the current pose of the target object in the visual coordinate system. The relative pose intermediate quantity determination module is used to calculate the relative pose intermediate quantity of the target object in the visual coordinate system based on the current pose and the keyframe pose in the visual coordinate system. The relative pose final quantity determination module is used to perform axis consistency processing on the relative pose intermediate quantity to output the relative pose final quantity in the control coordinate system.

8. A target tracking system based on arbitrary binocular images, characterized in that, include: First vision device, second vision device, and image processing device; Wherein, the first vision device is used to capture a first image of the target object; The second vision device is used to capture a second image of the target object; An image processing device is configured to acquire a first image and a second image of the current frame obtained by the first vision device and the second vision device simultaneously capturing images of a target object; perform image transpose processing on the first image and the second image to obtain a pseudo-horizontal binocular image, wherein the binocular baseline corresponding to the pseudo-horizontal binocular image is in the horizontal direction; input the pseudo-horizontal binocular image to the Simultaneous Localization and Mapping (SLAM) algorithm tracking module to output the current pose of the target object in the visual coordinate system; calculate the relative pose intermediate value of the target object in the visual coordinate system based on the current pose in the visual coordinate system and the keyframe pose; perform axis-consistent processing on the relative pose intermediate value to output the final relative pose value in the control coordinate system.

9. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the target tracking method based on any one of claims 1 to 6 via executing the executable instructions.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the target tracking method based on any one of claims 1 to 6.

11. A computer program product, comprising: A computer program or instruction, characterized in that, when executed by a processor, the computer program or instruction implements the target tracking method based on any one of claims 1 to 6.

Images