Visual multi-target tracking method based on panoramic image

Abstract

The invention discloses a visual multi-target tracking method based on a panoramic image, and belongs to the technical field of computer vision. The method comprises the following steps: inputting a panoramic RGB image into a feature extraction network, extracting image features and generating a learnable instance, and obtaining a bounding box and attribute information of a target through a decoding network; constructing a trajectory instance based on a historical trajectory to extract prediction information of a current frame, and in combination with a trajectory pool management strategy, realizing initialization, updating or deletion of the trajectory through confidence threshold judgment; designing a distance calculation and probability matching algorithm, fusing observation data of a learnable instance and a track instance, and optimizing a joint probability model associated with a target; an association mode is dynamically selected according to trajectory stability and environmental noise, and continuous online tracking and unique number management of multiple targets are ensured to be realized in a full scene. According to the method, the precision and robustness of multi-target tracking in a complex scene are effectively improved through high coverage of the panoramic image and an efficient trajectory association mechanism.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and in particular relates to a visual multi-target tracking method based on panoramic images. Background Art

[0002] In the field of computer vision, multi-target tracking technology is widely used in scenarios such as intelligent surveillance, autonomous driving, and robotic navigation. Traditional multi-target tracking methods are often based on images with limited field of view acquired by conventional cameras, achieving target tracking through steps such as feature extraction, data association, and trajectory prediction. However, the limited field of view of conventional cameras makes it difficult to cover a 360-degree panoramic environment, leading to problems such as target loss and tracking interruption in complex and dynamic scenes. In particular, in scenarios with densely populated targets, frequent occlusions, or rapid motion, the robustness and real-time performance of existing methods are significantly reduced.

[0003] In recent years, the widespread use of panoramic cameras has provided a new approach to wide-area scene perception. Panoramic images can capture complete information about the surrounding environment, theoretically improving the coverage capabilities of multi-target tracking. However, existing multi-target tracking technologies based on panoramic images still face many challenges: Image distortion and resolution difference: Panoramic images usually have spherical or cylindrical projection distortion, which makes it difficult to represent the target shape and motion direction. Traditional feature extraction networks are difficult to adapt to this nonlinear deformation.

[0004] Boundary crossing problem: The continuous movement of targets at the boundaries of panoramic images (such as stitching areas) can easily be misjudged as disappearance or new additions of targets. Existing algorithms lack effective boundary processing mechanisms.

[0005] High computational complexity: Panoramic images contain more targets. The computational complexity of traditional data association algorithms (such as the Hungarian algorithm) increases exponentially with the number of targets, making it difficult to meet real-time requirements.

[0006] Unintelligent trajectory management: Existing methods rely on fixed thresholds during trajectory initialization, update, and deletion, and cannot adapt to environmental changes (such as sudden changes in illumination and noise interference), which can easily lead to mistracking or waste of resources.

[0007] To address these issues, existing research has attempted to improve performance by refining feature representations or optimizing matching strategies. For example, some methods incorporate geometric transformations to mitigate distortion, but fail to address the global optimization problem of multi-target association. Other approaches employ Kalman or particle filters to predict target motion, but these methods are limited in effectiveness in panoramic scenes due to the increased complexity of the motion model. Furthermore, existing technologies inadequately integrate semantic features with physical information (such as velocity and acceleration), resulting in frequent target identity switching and poor stability, particularly during long-term tracking. Summary of the Invention

[0008] The purpose of an embodiment of the present invention is to provide a visual multi-target tracking method based on panoramic images. In response to the special needs of panoramic image multi-target tracking, it achieves high-precision, low-latency continuous tracking through dual-path feature extraction of learnable instances and trajectory instances, combined with an adaptive trajectory management strategy. This technology effectively overcomes the distortion interference of panoramic images, solves the problem of cross-border target tracking through boundary continuity transformation and intelligent matching algorithms, and optimizes data association using a joint probability model, significantly reducing computational complexity. In addition, the introduction of a dynamic threshold mechanism and semantic-physical feature fusion further enhances the robustness of the system in complex environments, providing a reliable technical foundation for panoramic perception in fields such as intelligent monitoring and autonomous driving, thereby resolving at least one technical problem involved in the background technology.

[0009] In order to solve the above-mentioned technical problems, the present invention is achieved as follows: The embodiment of the present invention provides a method for visual multi-target tracking based on panoramic images, comprising the following steps: Step S1: Input the panoramic RGB image into a feature extraction network to extract image features and generate a learnable instance; the learnable instance is processed by a decoding network to obtain the target's bounding box and attribute information, the attribute information including confidence, speed, motion direction, orientation, and acceleration; Step S2: If there are historical trajectories in the trajectory pool, a trajectory instance is constructed based on the historical trajectories. The trajectory instance obtains the bounding box and attribute information of the corresponding trajectory in the current frame through the decoding network; Step S3, performing trajectory management based on the output results of the learnable instance and the trajectory instance, specifically includes: If the confidence of the trajectory instance is higher than the update threshold , then update the corresponding trajectory; If the confidence of the trajectory instance is lower than the deletion threshold , then delete the corresponding trajectory from the trajectory pool; If the confidence of the learnable instance is higher than the initialization threshold , then initialize the new trajectory in the trajectory pool; Step S4: When data association is selected, the matching distances between the learnable instance, the trajectory instance, and all the trajectories in the trajectory pool are calculated, the association is completed through the matching algorithm, and the trajectory pool is updated according to the association results; In step S5, the trajectory management mode of step S3 or step S4 is selected through an adaptive strategy, and a unique number is assigned to the target to achieve continuous online tracking of multiple targets in a panoramic image.

[0010] Optionally, in step S2, the learnable instance and the trajectory instance both contain semantic features and physical information; Among the semantic features, the semantic features of the learnable instance come from the image features, and the semantic features of the trajectory instance come from the historical frame features stored in the trajectory pool; The physical information includes position, velocity, direction of motion, and acceleration, where the position information is represented by a panoramic continuity transform, and the target bounding box is defined as: ; Where, is the normalized upper left corner coordinate; is the normalized length position and ; is the normalized height position and ; is a symbolic function; is a sine function; , are width and height respectively; Indicates whether the bounding box crosses the panoramic image boundary.

[0011] Optionally, the semantic features and physical information of the trajectory instance are fed back to the sensor to quickly locate the target within the panoramic range and establish temporal association, thereby reducing matching complexity.

[0012] Optionally, in step S4, the calculation formula for the matching distance is: ; Where, and The confidence scores of bounding box 1 and bounding box 2 respectively; and For border crossing signs, Indicates that the border has not been crossed. Indicates crossing the border, Indicates that the border has not been crossed. Indicates crossing a boundary; is the adjustment factor; is the Mahalanobis distance of the eigenvector; is the normalization hyperparameter.

[0013] Optionally, in step S4, the matching algorithm is implemented by maximizing the joint probability model: ; in, is the trajectory instance observation set; is the set of learnable instance observations; represents the trajectory, and its subscripts 1, 2, ..., n represent the number of the trajectory; represents the number of trajectories; represents the number of trajectory instance observation sets; represents the number of learnable instance observation sets; Indicates on the trajectory Under the conditions, it was observed Probability of occurrence; Indicates on the trajectory Under the conditions, it was observed Probability of occurrence; by optimizing the association pattern The matching result that maximizes the joint probability is selected.

[0014] Optionally, in step S5, the adaptive strategy includes: Determine trajectory stability: If the historical update frequency of the trajectory And the maximum association probability Below threshold , then adopt the trajectory management of step S3; Judging environmental noise: If the number of random observations And the average association probability Below threshold , then the trajectory management of step S3 is adopted for all trajectories, where Indicates the maximum threshold of the number of observations; By default, the remaining trajectories adopt the data association management of step S4.

[0015] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention can achieve 360° coverage of the environment without blind spots through a single acquisition of panoramic cameras, avoiding the deployment complexity and high cost of traditional multi-camera systems while eliminating tracking blind spots. It is particularly suitable for real-time monitoring needs in large-scale scenarios (such as transportation hubs and smart security).

[0016] 2. Based on the collaborative design of feature extraction network and decoding network, the present invention can accurately extract the target's bounding box, confidence, speed, direction of movement and other attribute information from panoramic images. Combined with the dynamic generation mechanism of trajectory instances and learnable instances, it significantly improves the accuracy of target positioning and attribute analysis.

[0017] 3. This invention utilizes intelligent track pool management strategies (initialization, update, and deletion) and adaptive threshold control to effectively address complex scenarios such as target occlusion, temporary disappearance, or the appearance of new targets, ensuring tracking continuity and stability. Furthermore, by leveraging panoramic continuity transformation and boundary marker design within the physical information to address the characteristic of targets crossing boundaries in panoramic images, the risk of tracking loss due to these crossing boundaries is significantly reduced.

[0018] 4. This invention uses a composite distance metric formula that integrates confidence, boundary markers, and Mahalanobis distance, combined with a joint probability model for multi-target matching, significantly reducing the mismatch rate. An adaptive strategy dynamically selects the management method (end-to-end tracking or data association) based on trajectory stability and environmental noise, maintaining high robustness in complex scenarios.

[0019] 5. Through lightweight network design and optimization of matching algorithms, the present invention reduces computational complexity while ensuring high precision, meeting the needs of real-time online tracking. It can be widely used in fields with high timeliness requirements such as autonomous driving, robot navigation, and smart cities.

[0020] 6. The system provided by this invention can adapt to highly dynamic and high-noise environments (such as dense crowds and complex traffic flows), and dynamically adjusts management strategies through environmental noise assessment and trajectory stability judgment. It has strong generalization ability and practical value.

[0021] In general, the present invention achieves a comprehensive improvement in the accuracy, stability and real-time performance of multi-target tracking through the deep integration of panoramic image processing and intelligent algorithms, providing efficient and reliable technical support for smart cities, intelligent transportation and other fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. Those skilled in the art can also derive other drawings based on these drawings without inventive work, among which: Figure 1 A flowchart of a method for visual multi-target tracking based on panoramic images provided by an embodiment of the present invention; Figure 2 This is one of the hardware structure diagrams of the electronic device provided by the embodiment of the present invention; Figure 3 This is the second schematic diagram of the hardware structure of the electronic device provided by the embodiment of the present invention. DETAILED DESCRIPTION

[0023] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0024] The terms "first," "second," and the like in the specification and claims of the present invention are used to distinguish similar objects, and are not used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of the present invention can be implemented in an order other than that illustrated or described herein, and that the objects distinguished by "first," "second," and the like are generally of the same type, and do not limit the number of objects. For example, the first object can be one or more. In addition, the term "and / or" in the specification and claims refers to at least one of the connected objects, and the character " / " generally indicates that the objects connected are in an "or" relationship.

[0025] See Figure 1 As shown, an embodiment of the present invention provides a visual multi-target tracking method based on panoramic images, comprising the following steps: Step S1: Input the panoramic RGB image into a feature extraction network to extract image features and generate a learnable instance; the learnable instance is processed by a decoding network to obtain the target's bounding box and attribute information, the attribute information including confidence, speed, motion direction, orientation, and acceleration; Step S2: If there are historical trajectories in the trajectory pool, a trajectory instance is constructed based on the historical trajectories. The trajectory instance obtains the bounding box and attribute information of the corresponding trajectory in the current frame through the decoding network; Step S3, performing trajectory management based on the output results of the learnable instance and the trajectory instance, specifically includes: If the confidence of the trajectory instance is higher than the update threshold , then update the corresponding trajectory; If the confidence of the trajectory instance is lower than the deletion threshold , then delete the corresponding trajectory from the trajectory pool; If the confidence of the learnable instance is higher than the initialization threshold , then initialize the new trajectory in the trajectory pool; Step S4: When data association is selected, the matching distances between the learnable instance, the trajectory instance, and all the trajectories in the trajectory pool are calculated, the association is completed through the matching algorithm, and the trajectory pool is updated according to the association results; In step S5, the trajectory management mode of step S3 or step S4 is selected through an adaptive strategy, and a unique number is assigned to the target to achieve continuous online tracking of multiple targets in a panoramic image.

[0026] In step S2, the learnable instance and the trajectory instance both contain semantic features and physical information; Among the semantic features, the semantic features of the learnable instance come from the image features, and the semantic features of the trajectory instance come from the historical frame features stored in the trajectory pool; The physical information includes position, velocity, direction of motion, and acceleration, where the position information is represented by a panoramic continuity transform, and the target bounding box is defined as: ; Where, is the normalized upper left corner coordinate; is the normalized length position and ; is the normalized height position and ; is a symbolic function; is a sine function; , are width and height respectively; Indicates whether the bounding box crosses the panoramic image boundary.

[0027] The semantic features and physical information of the trajectory instance are fed back to the sensor to quickly locate the target in a panoramic range and establish temporal association, thereby reducing the matching complexity.

[0028] In step S4, the calculation formula of the matching distance is: ; Where, and The confidence scores of bounding box 1 and bounding box 2 respectively; and For border crossing signs, Indicates that the border has not been crossed. Indicates crossing the border, Indicates that the border has not been crossed. Indicates crossing a boundary; is the adjustment factor; is the Mahalanobis distance of the eigenvector; is the normalization hyperparameter.

[0029] The matching algorithm implements the matching strategy, including: Assume there are n trajectories: And two types of observation sets: Trajectory instance observation set: Learnable instance observation set: For the trajectory instance observation set and trajectory The association probability between can be expressed as follows: ; Where, It's a trajectory and trajectory instance observation set The residual between represents the dimension of the observation; is the covariance matrix.

[0030] For the learning instance observation set and trajectory The association probability between them is expressed as follows: ; Where, is a smaller coefficient, indicating that the association probability of random observations is low, avoiding false matches with existing trajectories.

[0031] The matching algorithm considers both the trajectory instance observation set and the learnable instance observation set, and is implemented by maximizing the joint probability model: ; in, is the trajectory instance observation set; is the set of learnable instance observations; represents the trajectory, and its subscripts 1, 2, ..., n represent the number of the trajectory; represents the number of trajectories; represents the number of trajectory instance observation sets; represents the number of learnable instance observation sets; Indicates on the trajectory Under the conditions, it was observed Probability of occurrence; Indicates on the trajectory Under the conditions, it was observed Probability of occurrence; Part 1 Calculate the probability of association between trajectory instance observations and trajectories, which is usually large; Part II Calculate the association probability between learning instance observations and trajectories, which is usually small; by optimizing the association pattern Select the matching result that maximizes the joint probability, which is expressed as follows: ; The optimization process maximizes the total joint probability by selecting the best association pattern, where Represents trajectory and observation Match, otherwise it is 0; is the indicator function, defined as follows: .

[0032] In step S5, the adaptive strategy includes: Determine trajectory stability: If the historical update frequency of the trajectory And the maximum association probability Below threshold , then adopt the trajectory management of step S3; Judging environmental noise: If the number of random observations And the average association probability Below threshold , then the trajectory management of step S3 is adopted for all trajectories, where Indicates the maximum threshold of the number of observations; By default, the remaining trajectories adopt the data association management of step S4.

[0033] See also Figure 2 As shown, an embodiment of the present invention further provides an electronic device 600, which includes a processor 601, a memory 602, and a program or instruction stored in the memory 602 and executable on the processor 601. When the program or instruction is executed by the processor 601, each process of the above-mentioned embodiment of the visual multi-target tracking method based on panoramic images is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0034] It should be noted that the electronic devices in the embodiments of the present invention include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0035] Figure 3 The present invention is a hardware structure diagram of an electronic device.

[0036] The electronic device 700 includes but is not limited to components such as a radio frequency unit 701, a network module 702, an audio output unit 703, an input unit 704, a sensor 705, a display unit 706, a user input unit 707, an interface unit 708, a memory 709, and a processor 710.

[0037] Those skilled in the art will understand that the electronic device 700 may also include a power supply (such as a battery) to power each component. The power supply may be logically connected to the processor 710 through a power management system, thereby implementing functions such as charging, discharging, and power consumption management through the power management system. Figure 3 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be repeated here.

[0038] It should be understood that in this embodiment of the present invention, the input unit 704 may include a graphics processing unit (GPU) 7041 and a microphone 7042. The GPU 7041 processes image data of still images or videos captured by an image capture system (e.g., a camera) in video capture mode or image capture mode. The display unit 706 may include a display panel 7061, which may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like. The user input unit 707 includes a touch panel 7071 and other input devices 7072. The touch panel 7071, also known as a touch screen, may include a touch detection system and a touch controller. Other input devices 7072 may include, but are not limited to, a physical keyboard, function keys (such as volume control buttons and power buttons), a trackball, a mouse, and a joystick, which are not described in detail here. The memory 709 may be used to store software programs and various data, including, but not limited to, applications and an operating system. The processor 710 may integrate an application processor and a modem processor. The application processor primarily processes the operating system, user interface, and applications, while the modem processor primarily handles wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 710.

[0039] An embodiment of the present invention also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned embodiment of the visual multi-target tracking method based on panoramic images are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0040] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer-readable storage medium, such as a computer read-only memory (ROM), random access memory (RAM), a magnetic disk, or an optical disk.

[0041] An embodiment of the present invention further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned embodiment of the visual multi-target tracking method based on panoramic images, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0042] It should be understood that the chip mentioned in the embodiment of the present invention can also be called a system-on-chip, a system-on-chip, a chip system, or a system-on-chip chip, etc.

[0043] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or system. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or system comprising the element.

[0044] Furthermore, it should be noted that the scope of the methods and systems of the present invention is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in reverse order, depending on the functions involved. For example, the methods described may be performed in an order different from that described, and various steps may be added, omitted, or combined. Furthermore, features described with reference to certain examples may be combined in other examples.

[0045] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are protected by the present invention.

Claims

1. A visual multi-target tracking method based on panoramic images, characterized in that: The steps include: Step S1: Input the panoramic RGB image into a feature extraction network to extract image features and generate a learnable instance; the learnable instance is processed by a decoding network to obtain the target's bounding box and attribute information, the attribute information including confidence, speed, motion direction, orientation, and acceleration; Step S2: If there are historical trajectories in the trajectory pool, a trajectory instance is constructed based on the historical trajectories. The trajectory instance obtains the bounding box and attribute information of the corresponding trajectory in the current frame through the decoding network; Step S3, performing trajectory management based on the output results of the learnable instance and the trajectory instance, specifically includes: If the confidence of the trajectory instance is higher than the update threshold , then update the corresponding trajectory; If the confidence of the trajectory instance is lower than the deletion threshold , then delete the corresponding trajectory from the trajectory pool; If the confidence of the learnable instance is higher than the initialization threshold , then initialize the new trajectory in the trajectory pool; Step S4: When data association is selected, the matching distances between the learnable instance, the trajectory instance, and all the trajectories in the trajectory pool are calculated, the association is completed through the matching algorithm, and the trajectory pool is updated according to the association results; In step S5, the trajectory management mode of step S3 or step S4 is selected through an adaptive strategy, and a unique number is assigned to the target to achieve continuous online tracking of multiple targets in a panoramic image.

2. The method according to claim 1, characterized in that In step S2, the learnable instance and the trajectory instance both contain semantic features and physical information; Among the semantic features, the semantic features of the learnable instance come from the image features, and the semantic features of the trajectory instance come from the historical frame features stored in the trajectory pool; The physical information includes position, velocity, direction of motion, and acceleration, where the position information is represented by a panoramic continuity transform, and the target bounding box is defined as: ； Where, is the normalized upper left corner coordinate; is the normalized length position and ; is the normalized height position and ; is a symbolic function; is a sine function; , are width and height respectively; Indicates whether the bounding box crosses the panoramic image boundary.

3. The method according to claim 2, characterized in that The semantic features and physical information of the trajectory instance are fed back to the sensor to quickly locate the target in a panoramic range and establish temporal association, thereby reducing the matching complexity.

4. The method according to claim 1, wherein In step S4, the calculation formula of the matching distance is: ； Where, and The confidence scores of bounding box 1 and bounding box 2 respectively; and For border crossing signs, Indicates that the border has not been crossed. Indicates crossing the border, Indicates that the border has not been crossed. It means crossing a boundary; is the adjustment factor; is the Mahalanobis distance of the eigenvector; is the normalization hyperparameter.

5. The method according to claim 1, wherein In step S4, the matching algorithm is implemented by maximizing the joint probability model: ； in, is the trajectory instance observation set; is the set of learnable instance observations; represents the trajectory, and its subscripts 1, 2, ..., n represent the number of the trajectory; represents the number of trajectories; represents the number of trajectory instance observation sets; represents the number of learnable instance observation sets; Indicates on the trajectory Under the conditions, it was observed Probability of occurrence; Indicates on the trajectory Under the conditions, it was observed Probability of occurrence; by optimizing the association pattern The matching result that maximizes the joint probability is selected.

6. The method according to claim 1, characterized in that In step S5, the adaptive strategy includes: Determine trajectory stability: If the historical update frequency of the trajectory And the maximum association probability Below threshold , then adopt the trajectory management of step S3; Judging environmental noise: If the number of random observations And the average association probability Below threshold , then the trajectory management of step S3 is adopted for all trajectories, where Indicates the maximum threshold of the number of observations; By default, the remaining trajectories adopt the data association management of step S4.

Images