A wide-scene multi-target adaptive cooperative tracking method and processing platform

Abstract

This invention discloses a wide-scene multi-target adaptive cooperative tracking method and processing platform for image monitoring and automatic tracking of multiple targets within the optical field of view in sea, land, and air backgrounds. The method includes the following steps: 1) acquiring image information from multiple optical sensors; 2) performing background consistency checks on the multiple images; 3) selecting the target to be tracked and calculating the target's starting position, image region, and feature vector; 4) initializing the processing platform's computing resource pool; 5) establishing a multi-target adaptive cooperative tracking model; and 6) performing high-precision and stable tracking of the selected target and outputting the current target tracking position. The processing platform consists of an embedded video processing board, a CPCI data communication board, and a structural housing.

Description

Technical Field

[0001] This invention relates to a collaborative tracking method and processing platform, and more particularly to a wide-scene multi-target adaptive collaborative tracking method and processing platform. Background Technology

[0002] Target tracking is a key technology in early warning detection and precision guidance, especially for multi-target tracking. Due to factors such as complex backgrounds and high computational demands, the accuracy and stability of target tracking face significant challenges, making it a research focus and hot topic for scholars both domestically and internationally. Classical target tracking methods mainly include template matching, optical flow, and inter-frame correlation. Template matching methods have high computational demands, affecting target tracking speed; optical flow methods have poor robustness against strong clutter backgrounds; and inter-frame correlation methods have low tracking accuracy for infrared targets with low signal-to-noise ratios. A non-parametric estimation mean-drift tracking algorithm disclosed in existing technology has good noise resistance and is insensitive to target rotation and deformation, making it suitable for infrared target tracking. Existing target appearance description-based target tracking methods and tracking algorithms combining motion and color features do not fundamentally solve the problem of interference from similar color backgrounds. The existing target tracking methods and processing platforms have the following drawbacks: (1) For wide-scene image tracking with multiple optical sensors, there is no efficient and stable multi-target tracking scheme and processing platform; (2) Traditional video processing platforms cannot meet the computing power requirements of high-precision multi-target tracking algorithms; (3) Traditional tracking algorithms cannot solve the problem of cross-boundary continuous tracking of targets between adjacent images. Summary of the Invention

[0003] Purpose of the invention: The technical problem to be solved by the present invention is to provide a wide-scene multi-target adaptive cooperative tracking method and processing platform to address the shortcomings of the existing technology.

[0004] To address the aforementioned technical problems, this invention discloses a wide-scene multi-target adaptive cooperative tracking method and processing platform, comprising the following steps:

[0005] Step 1: Acquire image information from one or more optical sensors to obtain multiple images;

[0006] Step 2: Perform a background consistency check on the multiple images;

[0007] Step 3: Select n targets to be tracked: Obj1, Obj2, ..., Obj n Calculate the starting position, image region, and feature vector for each target;

[0008] Step 4: Initialize the computing resource pool of the wide-scene multi-target adaptive collaborative tracking processing platform;

[0009] Step 5: Establish a multi-target adaptive cooperative tracking model;

[0010] Step 6: Track the specified target and output the tracking position of the specified target to complete wide-scene multi-target adaptive collaborative tracking.

[0011] A wide-scene multi-target adaptive collaborative tracking processing platform includes: an embedded video processing board, a CPCI data communication baseboard, and a structural box; wherein, the embedded video processing board is used to complete target tracking; the CPCI data communication baseboard is used to complete data interaction between video processing boards; and the structural box is used to install the embedded video processing board and the CPCI data communication baseboard.

[0012] The computing resource pool consists of two or more plug-and-play embedded video processing boards. Each embedded video processing board operates independently, and the embedded video processing boards are deployed in a stack manner.

[0013] Beneficial effects:

[0014] (1) It provides a complete solution for multi-channel optical sensor image tracking and video processing platform to achieve efficient and stable tracking of multiple targets in a wide scene;

[0015] (2) Using an embedded video processing board provides strong real-time performance and greatly improves computing efficiency. Through multi-platform collaborative processing, the problem of insufficient computing power of a single video processing board is solved.

[0016] (3) Effectively utilize the data communication interface of the embedded video processing board to perform data interaction of target position and target feature vector across multiple video processing boards, fundamentally realizing cross-boundary and relay tracking of the target between adjacent images. Attached Figure Description

[0017] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, and the advantages of the present invention in the above and / or other aspects will become clearer.

[0018] Figure 1 A schematic diagram of the workflow of this invention.

[0019] Figure 2 A diagram illustrating the collaborative operation of the computing resource pool.

[0020] Figure 3 Schematic diagram of an embedded image processing platform.

[0021] Figure 4 A schematic diagram of the display and control interface, images, and tracking results. Detailed Implementation

[0022] This invention provides a wide-scene multi-target adaptive cooperative tracking method and processing platform for target tracking of images from multiple optical sensors, providing wide-scene surveillance images and target tracking trajectories to meet user needs for target surveillance in sea, land, and air environments.

[0023] like Figure 1 As shown, the multi-target adaptive cooperative tracking method of the present invention mainly includes: acquiring multi-channel image information; background consistency check; calculating the starting position, image region and feature vector of the target to be tracked; initializing the computing resource pool of the processing platform; establishing a multi-target adaptive cooperative tracking model; and performing high-precision and stable tracking of the target.

[0024] Step 1: Acquire image information from multiple optical sensors;

[0025] Multi-channel optical sensors refer to cameras with ≥2 channels of visible light or infrared thermal imagers; image information refers to visible light images or infrared images.

[0026] Visible light images refer to high-definition (2K) or ultra-high-definition (4K) color images; infrared images refer to grayscale images of 320×256, 620×512, or 1024×1024.

[0027] Step 2: Perform a background consistency check on the multiple images;

[0028] Step 2-1: The multiple images are represented as Img1, Img2, ..., Img k The image Img1 ∪Img2 ∪…∪Img was calculated using the maximum inter-class variance method (reference: Otsu, Otsu NA Threshold Selection Method from Gray-Level Histograms[J].IEEE Transactions on Systems, Man, and Cybernetics, 1979, 9(1): 62-66.). k The segmentation threshold Th for foreground and background;

[0029] Step 2-2: Calculate the background value Img for each of the k images. 1，bac =∑(Img1(i,j)<Th),Img 2，bac =∑(Img2(i,j)<Th),...,Img k，bac =∑(Img k (i, j) < Th), and calculate Img. 1，bac 1mg 2，bac ..., Img k，bac Pixel mean Avg 1，bac Avg 2，bac...,Avg k，bac , where (i, j) is the pixel position;

[0030] Steps 2-3: Calculate the multi-path image Img1 ∪ Img2 ∪…∪ Img k Background mean Avg = (Avg 1，bac +Avg 2，bac +…+Avg k，bac ) / k;

[0031] Steps 2-4: Images Img'1, Img'2, ..., Img' after background consistency check k They are respectively Img'1 = Img1 + Avg - Avg 1，bac Img'2=Img2+Avg-Avg 2，bac ..., Img k =Img k +Avg-Avg k，bac .

[0032] Step 3: Select the targets to be tracked: Obj1, Obj2, ..., Obj n Calculate the target's starting position, image region, and feature vector;

[0033] Step 3-1: Obj1, Obj2, ..., Obj n The starting positions are manually set as (p1, q1), (p2, q2), ..., (p... n q n );

[0034] Step 3-2: Obj1, Obj2, ..., Obj n The image region is a P×Q window centered at the starting position;

[0035] Step 3-3: Assume the center of the target region Obj1 is y1, and the pixel position of the target in the image is within {x}. i} i＝1…v The target model is obtained by performing feature modeling on the target to be tracked (Cheng, YZMeanShift, Mode seeking, and Clustering[J].IEEE Transactions on Pattern Analysis and Machine Intelligence, 1995, 17(08): 790-799). The feature vector FEA of the target model is then obtained. 1，u Represented as:

[0036]

[0037]

[0038] Where kk(·) is the contour function of the kernel function, h is the bandwidth of the kernel function, δ(·) is the Delta function, and the function b(x) i ) is located at x i The pixel-to-grayscale index mapping, C is the normalization coefficient, v represents the number of sample points in the target area, and u represents the grayscale index number.

[0039] Obj2, ..., Obj n Feature vector FEA 2，u FEA n，u Calculation method and FEA 1，u same.

[0040] Step 4: Initialize the computing resource pool of the processing platform;

[0041] like Figure 3 As shown, the processing platform's computing resource pool consists of multiple plug-and-play embedded video processing boards (DSP+FPGA). Each embedded video processing board can operate independently, and the multiple embedded video processing boards are deployed in a stacked manner. Initializing the processing platform's computing resource pool is used to determine the number of embedded video processing boards.

[0042] After the computing resource pool is initialized, its collaborative working diagram is as follows: Figure 2 As shown, firstly, the FPGA on the embedded video processing board receives the source image from the optical sensor; secondly, the FPGA transmits the image to the DSP via an external data bus, and the DSP on the current video processing board completes the target tracking work. When the target moves across the boundary between adjacent images, the target feature vector is transmitted from the current video processing board DSP to the processing board corresponding to the adjacent image. The video processing board DSP then realizes continuous tracking of the target; finally, the target tracking position information is output.

[0043] Step 5: Establish a multi-target adaptive cooperative tracking model;

[0044] Step 5-1: Count the number of embedded video processing boards in the computing resource pool of the processing platform;

[0045] Step 5-2: Number the embedded video processing board and place it in the computation stack;

[0046] Step 5-3: When tracking a specified target, detect the idle embedded video processing board in the computing stack, pass the target feature vector and image information to the idle embedded video processing board, and mark the current processing board as used.

[0047] Step 5-4: When the target moves across boundaries between adjacent images, the target feature vector is transferred from the current processing board to the processing board corresponding to the adjacent image to achieve continuous tracking of the target.

[0048] Step 6: Perform high-precision and stable tracking on the selected target and output the current target tracking position.

[0049] The selected target is tracked based on the mean drift theory, and the current target tracking position is output.

[0050] Step 6-1: Select target Obj t The current positions are (p t q t ), specifying the neighborhood as (p t q t A region centered at point W with radius W. For any pixel value f(x, y) within this region, (xp...) t ) 2 +(yq t ) 2 <W 2 The quantization level to which f(x, y) belongs is u t The corresponding membership probability values ​​are as follows:

[0051]

[0052] After calculating the membership probability of all pixels within the region, the probability distribution map of the target neighborhood is obtained.

[0053] Step 6-2, based on the probability distribution map, with (p t q t Centered on a target area with width W0 and height H0, the magnitude of a single drift can be calculated. The calculation method is as follows:

[0054]

[0055] in:

[0056]

[0057]

[0058] Step 6-3: Repeat step 6-2 until the drift amount is reached. Less than the specified threshold Treshold o At that time, the current center point position (p′) will be set. t ,q′ t The target location is output.

[0059] This invention also proposes a wide-scene multi-target adaptive collaborative tracking processing platform, comprising: an embedded video processing board, a CPCI data communication baseboard, and a structural box; wherein, the embedded video processing board is used to complete target tracking; the CPCI data communication baseboard is used to complete data interaction between video processing boards; and the structural box is used to install the embedded video processing board and the CPCI data communication baseboard.

[0060] The computing resource pool consists of two or more plug-and-play embedded video processing boards. Each embedded video processing board operates independently, and the embedded video processing boards are deployed in a stack manner.

[0061] Example:

[0062] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0063] like Figure 1 As shown, this invention discloses a wide-scene multi-target adaptive cooperative tracking method and processing platform, including the following steps:

[0064] (1) Acquire image information from multiple optical sensors;

[0065] Here, the tracking method of the present invention is specifically introduced using an 8-channel 620×512 infrared grayscale image as an example.

[0066] (2) Perform background consistency check on multiple images;

[0067] The 8 images are represented as Img1, Img2, ..., Img8. The segmentation threshold Th = 76 for the foreground and background of the 8 images is calculated by the maximum inter-class variance method.

[0068] Statistical analysis of background value Img from 8 video streams 1，bac =∑(Img1(i,j)<81),Img 2，bac =∑(Img2(i,j)<81),...,Img k，bac =∑(Img k (i,j)<81), Img was calculated. 1，bac 1mg 2，bac ..., Img k，bac Pixel mean Avg 1，bac =68、Avg 2，bac =74、Avg 3，bac =72、Avg 4，bac =66、Avg 5，bac =73、Avg 6，bac =64、Avg 7，bac =73、Avg 8，bac =69, where (i, j) is the pixel position;

[0069] Calculate the approximate mean value of the background of the 8-channel images, Avg = 70.

[0070] Steps 2-4: Images Img'1, Img'2, ..., Img' after background consistency check k They are Img'1=Img1+2, Img'2=Img2-4, Img'3=Img3-2, Img'4=Img4+4, Img'5=Img5-3, Img'6=Img6+6, Img'7=Img7-3, Img'8=Img8+1.

[0071] (3) Select the target to be tracked, Obj1, and calculate the target's starting position, image region, and feature vector;

[0072] Select Obj1 in the 5th image path. The starting position of this target is (257, 132).

[0073] The image region of Obj1 is a 40×40 window centered at the starting position (257, 132);

[0074] The center y1 of the target region Obj1 is (257, 132), and the pixel position of the target in the image is defined by {x...} i} i＝1…1600 This indicates that the feature vector FEA of the target model is... 1，u Represented as:

[0075]

[0076]

[0077] Where kk(·) is the contour function of the kernel function, h is the bandwidth of the kernel function, δ(·) is the Delta function, and the function b(x) i ) is located at x i The pixel-to-grayscale index mapping, C is the normalization coefficient, v represents the number of sample points in the target area, and u represents the grayscale index number.

[0078] (4) Initialize the computing resource pool of the processing platform;

[0079] The number of embedded video processing boards is set to 8.

[0080] (5) Establish a multi-target adaptive cooperative tracking model;

[0081] The statistical processing platform's computing resource pool contains eight embedded video processing boards (DSP+FPGA). The DSP uses TI's C6455 fixed-point product, and the FPGA uses ALTERA's Stratix series product.

[0082] Step 5-2: The embedded video processing boards are numbered 1 to 8 and placed into the computation stack;

[0083] Step 5-3: When tracking a specified target, detect the idle embedded video processing board in the computing stack, pass the target feature vector and image information to the idle embedded video processing board, and mark the current processing board as used.

[0084] Step 5-4: When the target moves across boundaries between adjacent images, the target feature vector is transferred from the current processing board to the processing board corresponding to the adjacent image to achieve continuous tracking of the target.

[0085] (6) Perform high-precision and stable tracking of the selected target and output the current target tracking position.

[0086] The selected target is tracked based on the mean drift theory, and the current target tracking position is output.

[0087] Step 6-1: Select target Obj t The current positions are (257, 132). The specified neighborhood is a region centered at (257, 132) with a radius of W = 300. For any pixel value f(x, y) within this region, (xp t ) 2 +(yq t ) 2 <W 2 For a pixel f(x, y) = RGB(223, 221, 23) with a quantization level of 800, the membership probability values ​​at its corresponding position are as follows:

[0088] A(x, y) = FEA t，800

[0089] After calculating the membership probability of all pixels within the region, the probability distribution map of the target neighborhood is obtained.

[0090] Based on the probability distribution map, with (257, 132) as the center, the target area with width W0 = 40 and height H0 = 40 is examined, and the magnitude of a single drift can be calculated. The calculation method is as follows:

[0091]

[0092] Then this iteration Greater than the specified threshold Treshold o =3, continue iterating. After 4 iterations, Less than the specified threshold Treshold o =3. At this point, the corresponding (p′) t ,q′t The value is (271, 124), and (271, 124) is output as the target position.

[0093] Display interface images and target tracking results, as follows Figure 4 As shown, the numbers in the first row of the top 8 frames, from left to right, are: Figure 4 (1) Figure 4 (2) Figure 4 (3) and Figure 4 (4), the numbers in the second row from left to right are: Figure 4 (5) Figure 4 (6) Figure 4 (7) and Figure 4 (8); The left image in the lower half is a... Figure 4 (5) is an enlarged view. The image on the right is the target detection result of the navigation radar.

[0094] In its specific implementation, this application provides a computer storage medium and a corresponding data processing unit. The computer storage medium is capable of storing a computer program, which, when executed by the data processing unit, can run the invention's content regarding a wide-scene multi-target adaptive cooperative tracking method and processing platform, as well as some or all of the steps in various embodiments. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0095] Those skilled in the art will clearly understand that the technical solutions in the embodiments of the present invention can be implemented using computer programs and their corresponding general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of computer programs, i.e., software products. These computer program software products can be stored in a storage medium and include several instructions to cause a device containing a data processing unit (which may be a personal computer, server, microcontroller, MUU, or network device, etc.) to execute the methods described in various embodiments or certain parts of the embodiments of the present invention.

[0096] This invention provides a method and approach for wide-scene multi-target adaptive cooperative tracking and processing platform. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.

Claims

1. A wide-scene multi-target adaptive cooperative tracking method, characterized in that, Includes the following steps: Step 1: Acquire image information from one or more optical sensors to obtain multiple images; Step 2: Perform a background consistency check on the multiple images; Step 3: Select n targets to be tracked: Obj1, Obj2, ..., Obj n Calculate the starting position, image region, and feature vector for each target; Step 4: Initialize the computing resource pool of the wide-scene multi-target adaptive collaborative tracking processing platform; Step 5: Establish a multi-target adaptive cooperative tracking model; Step 6: Track the specified target and output the tracking position of the specified target to complete wide-scene multi-target adaptive collaborative tracking; Step 2, which involves performing a background consistency check on multiple images, includes the following steps: Step 2-1: The multiple images are represented as Img1, Img2, ..., Img k , of which Img k This represents the k-th image, i.e., the k-th image; the image Img1∪Img2∪…∪Img is calculated using the Otsu's method. k The segmentation threshold Th for foreground and background; Step 2-2: Statistically calculate the background values Img of k images respectively 1,bac = ∑(Img1(i,j) < Th), Img 2,bac = ∑(Img2(i,j) < Th), …, Img k,bac = ∑(Img k (i,j) < Th), and calculate the pixel means Avg 1,bac , Img 2,bac , …, Img k,bac of Img 1,bac , Avg 2,bac , …, Avg k,bac , where Img k,bac represents the background value of the k-th image, and Img k (i,j) represents the k-th image in terms of pixel positions, and (i,j) are the pixel positions in each image; Steps 2-3: Calculate the multi-path image Img1∪Img2∪…∪Img k The pixel mean of the background value, Avg = (Avg 1,bac +Avg 2,bac +…+Avg k,bac ) / k; Step 2-4: After steps 2-1 to 2-3, images Img'1, Img'2, ..., Img' are obtained. k They are respectively Img'1 = Img1 + Avg - Avg 1,bac Img'2=Img2+Avg-Avg 2,bac ..., Img k =Img k +Avg-Avg k,bac ; Step 3, which involves calculating the starting position, image region, and feature vector for each target, includes the following steps: Step 3-1: Obj1, Obj2, ..., Obj n The starting positions are (p1, q1), (p2, q2), ..., (p... n ,q n ); Step 3-2: Obj1, Obj2, ..., Obj n The image region is a P×Q window centered at the starting position, where P is the length of the image region and Q is the width of the image region; Step 3-3: Assume the center of the area occupied by the target Obj1 to be tracked is y1, and the pixel positions of the target in the image area are defined by {x}. i } i＝1...v It means that, where x i Let represent the i-th pixel, and v represent the number of pixels in the target to be tracked. Feature modeling of the target to be tracked yields the target model, and the feature vector FEA of the target model is then obtained. 1,u Represented as: Where kk(·) is the contour function of the kernel function, h is the bandwidth of the kernel function, δ(·) is the Delta function, and the function b(x) i ) is located at x i The pixel-to-grayscale index mapping, where C is the normalization coefficient; The nth target to be tracked, Obj n Feature vector FEA n,u Calculation method and FEA 1,u Similarly, the feature vectors of all targets to be tracked are calculated.

2. The wide-scene multi-target adaptive cooperative tracking method according to claim 1, characterized in that, The initialization described in step 4 includes the following steps: Initialize the computing resource pool of the wide-scene multi-target adaptive collaborative tracking processing platform, that is, determine the number of embedded video processing boards according to the number of optical sensors described in step 1.

3. The wide-scene multi-target adaptive cooperative tracking method according to claim 2, characterized in that, Step 5, establishing a multi-target adaptive cooperative tracking model, includes the following steps: Step 5-1: Count the number of embedded video processing boards in the computing resource pool of the wide-scene multi-target adaptive collaborative tracking processing platform; Step 5-2: Number the embedded video processing board and place it in the computation stack; Step 5-3: When tracking a specified target, detect an idle embedded video processing board in the computation stack, pass the feature vector of the specified target and the image of the area where the target is located to the idle embedded video processing board, set it as the current processing board, and mark the current processing board as used. Step 5-4: When the specified target moves across the boundary between adjacent images, the feature vector of the specified target is transferred from the current processing board to the processing board corresponding to the adjacent image to complete the continuous tracking of the specified target.

4. The wide-scene multi-target adaptive cooperative tracking method according to claim 3, characterized in that, Step 6, which involves tracking the specified target based on mean shift theory and outputting the tracking position of the specified target, specifically includes the following steps: Step 6-1: Calculate the membership probability of each pixel in the specified neighborhood of the specified target to obtain the probability distribution map of the neighborhood of the specified target; For the specified target Obj with number t t The current position is (p t ,q t ), specifying the neighborhood as (p t ,q t A region centered at (x, y) with radius W; for any pixel (x, y) within the neighborhood, the value f(x, y) is: (x-p t ) 2 +(y-q t ) 2 <W 2 Where f(x,y) belongs to the quantization level u t The corresponding membership probability values ​​are as follows: After calculating the membership probability of all pixels in the neighborhood, the probability distribution map of the target neighborhood is obtained; Step 6-2: Based on the probability distribution map, calculate the offset center point position of the search area of ​​the specified target, and calculate the drift amount; Step 6-3: Repeat step 6-2 until the drift is less than the threshold, and output the current offset center point position as the tracking position of the specified target.

5. The wide-scene multi-target adaptive cooperative tracking method according to claim 4, characterized in that, The specific method for calculating the drift amount described in step 6-2 includes: With point (p) t ,q t Centered on a target area with width W0 and height H0, the drift amount in a single instance is calculated. The size is calculated as follows: in: Among them, the midpoint (p′) t ,q′ t () indicates the target position after a single drift.

6. The wide-scene multi-target adaptive cooperative tracking method according to claim 5, characterized in that, The multi-channel optical sensor mentioned in step 1 refers to a visible light camera or an infrared thermal imager with ≥2 channels; the image information refers to the information of the visible light image or the infrared image. Visible light images refer to high-definition (HD) color images with a resolution of 2K or ultra-high-definition (UHD) color images with a resolution of 4K; infrared images refer to grayscale images of 320×256, 620×512, or 1024×1024.

7. A wide-scene multi-target adaptive cooperative tracking processing platform, characterized in that, The wide-scene multi-target adaptive cooperative tracking method as described in claim 6 includes: an embedded video processing board, a CPCI data communication baseboard, and a structural box; wherein, the embedded video processing board is used to complete target tracking; the CPCI data communication baseboard is used to complete data interaction between video processing boards; and the structural box is used to install the embedded video processing board and the CPCI data communication baseboard.

8. The wide-scene multi-target adaptive cooperative tracking processing platform according to claim 7, characterized in that, The computing resource pool consists of two or more plug-and-play embedded video processing boards. Each embedded video processing board operates independently, and the embedded video processing boards are deployed in a stack manner.

Images