A visual multi-target tracking method based on panoramic images

Abstract

The application discloses a kind of visual multi-target tracking methods based on panoramic image, belong to computer vision technical field.The method is by inputting panoramic RGB image to feature extraction network, extracts image feature and generates learnable instance, and obtains the boundary box and attribute information of target by decoding network;Based on historical trajectory, trajectory instance is constructed to extract the prediction information of current frame, in combination with trajectory pool management strategy, the initialization, update or deletion of trajectory is realized by confidence threshold judgment;Design distance calculation and probability matching algorithm, the observation data of learnable instance and trajectory instance are fused, and the joint probability model of target association is optimized;According to the stability of trajectory and environmental noise, dynamically select association mode, ensure that continuous online tracking and unique numbering management of multi-target are realized in full scene.The method improves the precision and robustness of multi-target tracking in complex scene by the high coverage of panoramic image and efficient trajectory association mechanism.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a visual multi-target tracking method based on panoramic images. Background Technology

[0002] In the field of computer vision, multi-object tracking technology is widely used in scenarios such as intelligent monitoring, autonomous driving, and robot navigation. Traditional multi-object tracking methods are mostly based on images with limited fields of view acquired by ordinary cameras, achieving object tracking through steps such as feature extraction, data association, and trajectory prediction. However, the limited field of view of ordinary cameras makes it difficult to cover a 360-degree panoramic environment, leading to problems such as target loss and tracking interruption in complex and dynamic scenes. Especially in scenarios with dense targets, frequent occlusion, or rapid movement, the robustness and real-time performance of existing methods decrease significantly.

[0003] In recent years, the widespread adoption of panoramic cameras has provided a new approach to wide-area scene perception. Panoramic images can capture complete information about the surrounding environment, theoretically improving the coverage of multi-target tracking. However, existing multi-target tracking technologies based on panoramic images still face many challenges:

[0004] Image distortion and resolution differences: Panoramic images often have spherical or cylindrical projection distortion, which makes it difficult to represent the shape and direction of motion of the target. Traditional feature extraction networks are difficult to adapt to this nonlinear deformation.

[0005] Boundary crossing problem: The continuous movement of a target at the boundary of a panoramic image (such as a stitching area) is easily misjudged as a disappearance or addition of a target, and existing algorithms lack an effective boundary processing mechanism.

[0006] High computational complexity: Panoramic images contain more targets, and the computational complexity of traditional data association algorithms (such as the Hungarian algorithm) increases exponentially with the number of targets, making it difficult to meet real-time requirements.

[0007] Trajectory management is not intelligent: Existing methods rely on fixed thresholds when initializing, updating and deleting trajectories, which cannot adapt to changes in the environment (such as sudden changes in lighting or noise interference), and are prone to mistracking or waste of resources.

[0008] To address these issues, existing research has attempted to improve performance by refining feature representations or optimizing matching strategies. For example, some methods introduce geometric transformations to mitigate distortion, but fail to solve the global optimization problem of multi-target association; other schemes use Kalman filtering or particle filtering to predict target motion, but their effectiveness is limited in panoramic scenes due to the increased complexity of the motion model. Furthermore, existing technologies lack sufficient fusion of semantic features and physical information (such as velocity and acceleration), leading to frequent target identity switching, especially resulting in poor stability during long-term tracking. Summary of the Invention

[0009] The purpose of this invention is to provide a visual multi-target tracking method based on panoramic images. Addressing the specific needs of multi-target tracking in panoramic images, this method achieves high-precision, low-latency continuous tracking through dual-path feature extraction of learnable instances and trajectory instances, combined with an adaptive trajectory management strategy. This technology effectively overcomes distortion interference in panoramic images, solves the challenge of target cross-boundary tracking through boundary continuity transformation and intelligent matching algorithms, and significantly reduces computational complexity by optimizing data association using a joint probability model. Furthermore, the introduction of a dynamic threshold mechanism and semantic-physical feature fusion further enhances the system's robustness in complex environments, providing a reliable technical foundation for panoramic perception in fields such as intelligent monitoring and autonomous driving, thereby solving at least one of the technical problems mentioned in the background art.

[0010] To solve the above-mentioned technical problems, the present invention is implemented as follows:

[0011] This invention provides a visual multi-target tracking method based on panoramic images, comprising the following steps:

[0012] Step S1: Input the panoramic RGB image into the feature extraction network to extract image features and generate learnable instances; the learnable instances are processed by the decoding network to obtain the bounding box and attribute information of the target, including confidence, velocity, motion direction, orientation and acceleration;

[0013] Step S2: If there are historical trajectories in the trajectory pool, then a trajectory instance is constructed based on the historical trajectory. The trajectory instance obtains the bounding box and attribute information of the corresponding trajectory in the current frame through the decoding network.

[0014] Step S3 involves trajectory management based on the output results of learnable instances and trajectory instances, specifically including:

[0015] If the confidence level of the trajectory instance is higher than the update threshold If so, then update the corresponding trajectory;

[0016] If the confidence level of a trajectory instance is lower than the deletion threshold If so, the corresponding trajectory is deleted from the trajectory pool;

[0017] If the confidence level of the learnable instance is higher than the initial threshold If so, a new trajectory is initialized in the trajectory pool;

[0018] Step S4: When data association is selected, calculate the matching distance between the learnable instance, the trajectory instance and all trajectories in the trajectory pool, complete the association through the matching algorithm, and update the trajectory pool according to the association results;

[0019] Step S5: Select the trajectory management method of step S3 or step S4 through an adaptive strategy, assign a unique number to the target, and realize continuous online tracking of multiple targets under panoramic images.

[0020] Optionally, in step S2, both the learnable instance and the trajectory instance contain semantic features and physical information;

[0021] In the semantic features, the semantic features of learnable instances come from image features, and the semantic features of trajectory instances come from historical frame features stored in the trajectory pool.

[0022] The physical information includes position, velocity, direction of motion, and acceleration, wherein the position information is represented by a panoramic continuity transform, and the target bounding box is defined as follows:

[0023] ;

[0024] In the formula, The coordinates of the top-left corner after normalization; The normalized length position and ; The normalized height position and ; It is a symbolic function; It is a sine function; , They are width and height, respectively; Indicate whether the bounding box crosses the boundary of the panoramic image.

[0025] Optionally, the semantic features and physical information of the trajectory instance are fed back to the perceptron to quickly locate the target and establish temporal association within the panoramic range, thereby reducing matching complexity.

[0026] Optionally, in step S4, the formula for calculating the matching distance is:

[0027] ;

[0028] In the formula, and These are the confidence scores for bounding box 1 and bounding box 2, respectively. and For border crossing markers, This indicates that the boundary has not been crossed. Indicates crossing a boundary. This indicates that the boundary has not been crossed. Indicates crossing a boundary; For adjustment factors; The Mahalanobis distance of the feature vectors; These are normalized hyperparameters.

[0029] Optionally, in step S4, the matching algorithm is implemented by maximizing the joint probability model:

[0030] ;

[0031] in, For trajectory instance observation set; For learnable instance observation set; The symbol represents the trajectory, and its subscripts 1, 2, ..., n indicate which current trajectory is in the current trajectory. Indicates the number of trajectories; Indicates the number of observation sets for trajectory instances; Indicates the number of learnable instance observations; Indicates in trajectory Under these conditions, it was observed The probability of occurrence; Indicates in trajectory Under these conditions, it was observed The probability of occurrence; by optimizing the association pattern Choose the matching result that maximizes the joint probability.

[0032] Optionally, in step S5, the adaptive strategy includes:

[0033] Determine trajectory stability: If the historical update frequency of the trajectory... And the maximum association probability Below the threshold Then, the trajectory management in step S3 is adopted;

[0034] Determine environmental noise: If the number of random observations And average association probability Below the threshold Then, trajectory management in step S3 is applied to all trajectories, where... This represents the maximum threshold for the number of observations.

[0035] By default: the remaining trajectories are managed using the data association method in step S4.

[0036] Compared with the prior art, the advantages of this invention are as follows:

[0037] 1. This invention can achieve 360° coverage of the environment without blind spots with a single acquisition by a panoramic camera, avoiding the deployment complexity and high cost of traditional multi-camera systems, while eliminating tracking blind spots. It is especially suitable for real-time monitoring needs in large-scale scenarios (such as transportation hubs and intelligent security).

[0038] 2. Based on the collaborative design of feature extraction network and decoding network, this invention can accurately extract the target's bounding box, confidence level, speed, motion direction and other attribute information from panoramic images. Combined with the dynamic generation mechanism of trajectory instances and learnable instances, it significantly improves the accuracy of target localization and attribute analysis.

[0039] 3. This invention effectively addresses complex scenarios such as target occlusion, temporary disappearance, or the appearance of new targets through intelligent management strategies (initialization, updating, and deletion) and adaptive threshold control of the trajectory pool, ensuring the continuity and stability of tracking. Furthermore, considering the characteristic of targets crossing boundaries in panoramic images, the invention significantly reduces the risk of tracking loss due to target boundary crossings through panoramic continuity transformation and boundary marker design in physical information.

[0040] 4. This invention employs a composite distance metric formula that integrates confidence level, boundary markers, and Mahalanobis distance, combined with a joint probability model for multi-target matching, significantly reducing the false matching rate. The adaptive strategy dynamically selects the management method (end-to-end tracking or data association) based on trajectory stability and environmental noise, maintaining high robustness even in complex scenarios.

[0041] 5. This invention reduces computational complexity while ensuring high accuracy through lightweight network design and optimization of matching algorithms, meeting the requirements of real-time online tracking. It can be widely used in fields with high timeliness requirements such as autonomous driving, robot navigation, and smart cities.

[0042] 6. The system provided by this invention can adapt to highly dynamic and high-noise environments (such as dense crowds and complex traffic flows), and dynamically adjusts management strategies through environmental noise assessment and trajectory stability judgment, possessing strong generalization ability and practical value.

[0043] In summary, this invention achieves a comprehensive improvement in the accuracy, stability, and real-time performance of multi-target tracking through the deep integration of panoramic image processing and intelligent algorithms, providing efficient and reliable technical support for fields such as smart cities and intelligent transportation. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort, wherein:

[0045] Figure 1 A flowchart illustrating a visual multi-target tracking method based on panoramic images provided in an embodiment of the present invention;

[0046] Figure 2This is one of the hardware structure diagrams of the electronic device provided in the embodiments of the present invention;

[0047] Figure 3 This is the second schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present invention. Detailed Implementation

[0048] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0049] The terms "first," "second," etc., used in the specification and claims of this invention are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0050] Please see Figure 1 As shown, this embodiment of the invention provides a visual multi-target tracking method based on panoramic images, including the following steps:

[0051] Step S1: Input the panoramic RGB image into the feature extraction network to extract image features and generate learnable instances; the learnable instances are processed by the decoding network to obtain the bounding box and attribute information of the target, including confidence, velocity, motion direction, orientation and acceleration;

[0052] Step S2: If there are historical trajectories in the trajectory pool, then a trajectory instance is constructed based on the historical trajectory. The trajectory instance obtains the bounding box and attribute information of the corresponding trajectory in the current frame through the decoding network.

[0053] Step S3 involves trajectory management based on the output results of learnable instances and trajectory instances, specifically including:

[0054] If the confidence level of the trajectory instance is higher than the update threshold If so, then update the corresponding trajectory;

[0055] If the confidence level of a trajectory instance is lower than the deletion threshold If so, the corresponding trajectory is deleted from the trajectory pool;

[0056] If the confidence level of the learnable instance is higher than the initial threshold If so, a new trajectory is initialized in the trajectory pool;

[0057] Step S4: When data association is selected, calculate the matching distance between the learnable instance, the trajectory instance and all trajectories in the trajectory pool, complete the association through the matching algorithm, and update the trajectory pool according to the association results;

[0058] Step S5: Select the trajectory management method of step S3 or step S4 through an adaptive strategy, assign a unique number to the target, and realize continuous online tracking of multiple targets under panoramic images.

[0059] In step S2, both the learnable instance and the trajectory instance contain semantic features and physical information;

[0060] In the semantic features, the semantic features of learnable instances come from image features, and the semantic features of trajectory instances come from historical frame features stored in the trajectory pool.

[0061] The physical information includes position, velocity, direction of motion, and acceleration, wherein the position information is represented by a panoramic continuity transform, and the target bounding box is defined as follows:

[0062] ;

[0063] In the formula, The coordinates of the top-left corner after normalization; The normalized length position and ; The normalized height position and ; It is a symbolic function; It is a sine function; , They are width and height, respectively; Indicate whether the bounding box crosses the boundary of the panoramic image.

[0064] The semantic features and physical information of the trajectory instance are fed back to the perceptron to quickly locate the target and establish temporal correlation within the panoramic range, thereby reducing matching complexity.

[0065] In step S4, the formula for calculating the matching distance is:

[0066] ;

[0067] In the formula, and These are the confidence scores for bounding box 1 and bounding box 2, respectively. and For border crossing markers, This indicates that the boundary has not been crossed. Indicates crossing a boundary. This indicates that the boundary has not been crossed. Indicates crossing a boundary; For adjustment factors; The Mahalanobis distance of the feature vectors; These are normalized hyperparameters.

[0068] The matching algorithm employs a matching strategy, which specifically includes:

[0069] Suppose there are n trajectories: And two types of observation sets:

[0070] Trajectory instance observation set:

[0071] Learnable instance observation set:

[0072] For trajectory instance observation set and trajectory The probability of association between them can be expressed by the following formula:

[0073] ;

[0074] In the formula, It is a trajectory and trajectory instance observation set The residuals between; Indicates the dimension of observation; It is the covariance matrix.

[0075] For the learning instance observation set and trajectory The probability of association between them is expressed by the following formula:

[0076] ;

[0077] In the formula, It is a small coefficient, indicating a low probability of association between random observations, thus avoiding incorrect matching with existing trajectories.

[0078] The matching algorithm considers both the trajectory instance observation set and the learnable instance observation set, and achieves this by maximizing the joint probability model:

[0079] ;

[0080] in, For trajectory instance observation set; For learnable instance observation set; The symbol represents the trajectory, and its subscripts 1, 2, ..., n indicate which current trajectory is in the current trajectory. Indicates the number of trajectories; Indicates the number of observation sets for trajectory instances; Indicates the number of learnable instance observations; Indicates in trajectory Under these conditions, it was observed The probability of occurrence; Indicates in trajectory Under these conditions, it was observed Probability of occurrence; Part 1 The probability of association between trajectory instance observations and trajectories is typically high; Part Two The probability of association between observations and trajectories in learning instances is typically small; this can be addressed by optimizing the association pattern. The matching result that maximizes the joint probability is chosen by the following formula:

[0081] ;

[0082] The optimization process maximizes the total joint probability by selecting the optimal association pattern, where Representing the trajectory and observation A match is given, otherwise 0; where It is an indicator function, defined as follows:

[0083] .

[0084] In step S5, the adaptive strategy includes:

[0085] Determine trajectory stability: If the historical update frequency of the trajectory... And the maximum association probability Below the threshold Then, the trajectory management in step S3 is adopted;

[0086] Determine environmental noise: If the number of random observations And average association probability Below the threshold Then, trajectory management in step S3 is applied to all trajectories, where... This represents the maximum threshold for the number of observations.

[0087] By default: the remaining trajectories are managed using the data association method in step S4.

[0088] See also Figure 2As shown, this embodiment of the invention also provides an electronic device 600, which includes a processor 601, a memory 602, and a program or instructions stored in the memory 602 and executable on the processor 601. When the program or instructions are executed by the processor 601, they implement the various processes of the above-described embodiments of the visual multi-target tracking method based on panoramic images and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0089] It should be noted that the electronic devices in the embodiments of the present invention include the mobile electronic devices and non-mobile electronic devices described above.

[0090] Figure 3 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present invention.

[0091] The electronic device 700 includes, but is not limited to, components such as: radio frequency unit 701, network module 702, audio output unit 703, input unit 704, sensor 705, display unit 706, user input unit 707, interface unit 708, memory 709, and processor 710.

[0092] Those skilled in the art will understand that the electronic device 700 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 710 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 3 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0093] It should be understood that, in this embodiment of the invention, the input unit 704 may include a graphics processing unit (GPU) 7041 and a microphone 7042. The GPU 7041 processes image data of still images or videos obtained by an image capture system (such as a camera) in video capture mode or image capture mode. The display unit 706 may include a display panel 7061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, etc. The user input unit 707 includes a touch panel 7071 and other input devices 7072. The touch panel 7071 is also called a touch screen. The touch panel 7071 may include a touch detection system and a touch controller. Other input devices 7072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be described in detail here. The memory 709 can be used to store software programs and various data, including but not limited to applications and operating systems. The processor 710 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understandable that the aforementioned modem processor may not be integrated into the processor 710.

[0094] This invention also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described embodiments of the visual multi-target tracking method based on panoramic images and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0095] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0096] This invention also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described embodiments of the visual multi-target tracking method based on panoramic images, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0097] It should be understood that the chip mentioned in the embodiments of the present invention may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0098] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0099] Furthermore, it should be noted that the scope of the methods and systems in the embodiments of the present invention is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. In addition, features described with reference to certain examples may be combined in other examples.

[0100] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of the present invention.

Claims

1. A visual multi-target tracking method based on panoramic images, characterized in that, Includes the following steps: Step S1: Input the panoramic RGB image into the feature extraction network to extract image features and generate learnable instances; the learnable instances are processed by the decoding network to obtain the bounding box and attribute information of the target, including confidence, velocity, motion direction, orientation and acceleration; Step S2: If there are historical trajectories in the trajectory pool, then a trajectory instance is constructed based on the historical trajectory. The trajectory instance obtains the bounding box and attribute information of the corresponding trajectory in the current frame through the decoding network. Step S3 involves trajectory management based on the output results of learnable instances and trajectory instances, specifically including: If the confidence level of a trajectory instance is higher than the update threshold θ update If so, then update the corresponding trajectory; If the confidence level of a trajectory instance is lower than the deletion threshold θ delete If so, the corresponding trajectory is deleted from the trajectory pool; If the confidence of the learnable instance is higher than the initial threshold θ initalize If so, a new trajectory is initialized in the trajectory pool; Step S4: When data association is selected, the matching distance between the learnable instance, the trajectory instance, and all trajectories in the trajectory pool is calculated. The association is completed using a matching algorithm, and the trajectory pool is updated based on the association results. The matching algorithm is implemented by maximizing the joint probability model. Among them, O h For trajectory instance observation set; O r The set of learnable instance observations; T represents the trajectory, and its subscripts 1, 2, ..., n indicate which trajectory is currently in existence; N Indicates the number of trajectories; K L represents the number of trajectory instance observations; L represents the number of learnable instance observations; p(O hj |T i ) indicates that on trajectory T i Under these conditions, O was observed hj The probability of occurrence; p(O) rj |T i ) indicates that on trajectory T i Under these conditions, O was observed rj The probability of occurrence; by optimizing the association pattern Choose the matching result that maximizes the joint probability; Step S5: Select the trajectory management method from either Step S3 or Step S4 using an adaptive strategy, assign a unique number to the target, and achieve continuous online tracking of multiple targets in the panoramic image. The adaptive strategy includes: Determine trajectory stability: If the historical update frequency of the trajectory is valid_ratio(T) i )≥α and the maximum association probabilitymax j P assoc (O rj ,T i (Below the threshold τ) p Then, the trajectory management in step S3 is adopted; Determine environmental noise: If the number of random observations |O r |>N max And average association probability Below the threshold τ q Then, trajectory management in step S3 is applied to all trajectories, where N max This represents the maximum threshold for the number of observations. By default: the remaining trajectories are managed using the data association method in step S4.

2. The method according to claim 1, characterized in that, In step S2, both the learnable instance and the trajectory instance contain semantic features and physical information; In the semantic features, the semantic features of learnable instances come from image features, and the semantic features of trajectory instances come from historical frame features stored in the trajectory pool. The physical information includes position, velocity, direction of motion, and acceleration, wherein the position information is represented by a panoramic continuity transform, and the target bounding box is defined as follows: (α,l,w,h,sgn(sin(α+w))); In the formula, (α,l) is the normalized top-left corner coordinate; α is the normalized length position and α∈[0,π]; l is the normalized height position and l∈[0,1]; sgn(·) is the sign function; sin(·) is the sine function; w and h are the width and height, respectively; sgn(sin(α+w)) indicates whether the bounding box crosses the panoramic image boundary.

3. The method according to claim 2, characterized in that, The semantic features and physical information of the trajectory instance are fed back to the perceptron to quickly locate the target and establish temporal correlation within the panoramic range, thereby reducing matching complexity.

4. The method according to claim 1, characterized in that, In step S4, the formula for calculating the matching distance is: In the formula, s1 and s2 are the confidence scores of bounding box 1 and bounding box 2, respectively; sgn1 and sgn2 are boundary crossing indicators, where sgn1>0 indicates no boundary crossing, sgn1<0 indicates boundary crossing, and sgn2>0 indicates no boundary crossing, sgn2<0 indicates boundary crossing; λ is the adjustment factor; D M (f1,f2) is the Mahalanobis distance between the eigenvectors; σ 2 These are normalized hyperparameters.

Images