A target tracking method suitable for fiber optic image bundle type bionic compound eye

Abstract

This invention discloses a target tracking method suitable for a fiber optic image bundle-type bionic compound eye. The method involves preprocessing a bionic compound eye image comprising multiple sub-images; identifying moving targets; extracting features from the moving targets to generate a coherent filter template; constructing a rotation matrix for the moving targets; and determining whether the moving targets have left the current sub-image: 1) If the moving targets have not left the current sub-image, the position of the moving targets is obtained based on the position of the response peak, which is used as the target tracking result; 2) If the moving targets have left the current sub-image, the three most likely inference sub-images where the moving targets appear are identified. Based on these three inference sub-images, rotation operations, feature extraction, and convolution calculations with the coherent filter template are performed to obtain a new position for the moving targets, which is used as the tracking position. Compared with existing technologies, this invention can achieve accurate tracking of moving targets within a large field of view.

Description

Technical Field

[0001] This invention relates to the fields of optical detection and optical imaging, and in particular to a target tracking method suitable for a bionic compound eye. Background Technology

[0002] Traditional target tracking systems are mainly divided into two types: monocular vision tracking systems and multi-view vision tracking systems. Monocular vision tracking systems have a small field of view. If fisheye cameras or rotating cameras are used to expand the field of view, there are drawbacks such as poor imaging at the edges of the field of view or poor real-time performance. Multi-view vision tracking systems are large and complex, making them difficult to set up in outdoor scenes. Therefore, achieving accurate target tracking over a large field of view has always been a major challenge in the field of target tracking.

[0003] Compared to traditional vision systems inspired by the human eye, curved bionic compound eye systems inspired by compound eyes offer advantages such as a large field of view, high resolution, and high motion sensitivity, making them highly promising for target tracking. Among the currently proposed curved bionic compound eye systems, fiber optic image bundle-based bionic compound eye systems (such as Chinese Patent CN117008245A, "A Bionic Compound Eye System Based on Fiber Optic Image Bundle") not only provide clear imaging but also feature a simple and compact structure. The images from each sub-eye are of good quality and uniformity, avoiding the problems of poor edge imaging and difficulty in real-time imaging, thus meeting the various requirements for imaging devices in the target tracking field.

[0004] Currently, target tracking algorithms based on coherent filtering have been a focus of research for many scholars due to their advantages of fast tracking speed and high tracking accuracy, as exemplified by Chinese patents CN118071792A ("A Method and System for Target Detection and Tracking of Unmanned Aerial Vehicles") and CN118071826A ("A Method and System for Target Localization Based on Video Surveillance"). Furthermore, target tracking algorithms based on coherent filtering, while ensuring high accuracy and tracking speed, also have relatively low requirements for computer configuration, making them well-suited for bionic compound eye systems.

[0005] Unlike traditional optical systems, images captured by a bionic compound eye system consist of numerous sub-images. When imaging a moving target, the target's image moves between these sub-images. Furthermore, the fiber optic image bundle type bionic compound eye system requires fixing the fiber optic image bundle connecting each sub-eye to a fiber optic bundle panel. Due to the inherent flexibility of the fiber optic image bundle, its rear end face rotates during fixing, resulting in varying degrees of rotation in the final sub-images. Because of these imaging characteristics, traditional target tracking algorithms cannot be directly applied to this type of bionic compound eye system. Summary of the Invention

[0006] This invention proposes a target tracking method applicable to fiber optic image bundle type bionic compound eye. It aims to utilize the imaging characteristics of fiber optic image bundle type bionic compound eye system, determine the relationship between moving target and sub-image based on response peak value, and obtain target tracking results when moving target moves between different sub-images.

[0007] To achieve the above-mentioned objectives, the present invention proposes the following technical solution:

[0008] A target tracking method suitable for fiber optic image bundle type bionic compound eye includes the following steps:

[0009] S1, preprocess the bionic compound eye image, which includes multiple sub-images, acquired by the compound eye system, to obtain the center coordinates and rotation angle of each sub-image;

[0010] S2, using a three-frame interpolation algorithm for moving target recognition, obtains the initial pixel position (xt0, yt0) of the moving target and the basic information of the initial sub-image where the moving target is located, including the center coordinates (x0, y0) and rotation angle b of the initial sub-image, and the relative rotation angle. Initialize to 0;

[0011] S3, extract features in the moving target region and generate a coherent filter template for the moving target;

[0012] S4, construct the rotation matrix T, as shown below:

[0013]

[0014] The next frame image is rotated using a rotation matrix T until the rotation angles of the current sub-image and the initial sub-image are consistent, thus obtaining the predicted region of the moving target. Features are extracted from the predicted region and convolved with a coherent filter template: f(z) = h*z, where f(z) is the obtained response result, h is the generated coherent filter template, and z is the feature result extracted in the predicted region. The response peak value and the position of the response peak value are recorded.

[0015] S5, perform the following operations sequentially based on the relationship between the response peak value η and the set threshold a, to determine whether the moving target has left the current sub-image in the prediction area;

[0016] If the response peak η is greater than the set threshold a, it indicates that the moving target has not left the current sub-image. The new position of the moving target is obtained based on the position of the response peak and used as the target tracking result.

[0017] If the peak response η is less than the set threshold a, it is determined that the moving target has left the current sub-image. Then, based on the positional relationship between other sub-images and the current sub-image, and the direction of motion of the moving target in the current sub-image,

[0018] N1, determine the three most likely inference sub-images of the moving target, obtain the center coordinates (xn, yn) and rotation angles b1, b2, b3 of the three inference sub-images, and then obtain the rotation angles θ1 = (b1-b), θ2 = (b2-b), θ3 = (b3-b) relative to the initial sub-image, collectively referred to as θn;

[0019] N2, construct the rotation matrix based on the rotation angles of the three inference sub-images relative to the initial sub-image, as shown in the following expression:

[0020]

[0021] Where θn is the rotation angle of the three inference sub-images relative to the initial sub-image, (xn,yn) is the center coordinate of the three inference sub-images, and n is the inference sub-image number, n = 1, 2, 3, until the rotation angle of the three inference sub-images is consistent with that of the initial sub-image, thus obtaining the inference region of the moving target;

[0022] N3, extract features from the inference region and perform convolution operation with the coherent filter template to obtain the response peaks of the three inference regions and the positions of the response peaks of the three inference regions, and take the maximum response peak.

[0023] N4 is used to perform convolution operations to determine the new position of the moving target.

[0024] xt=xn-ymax sin(bn-b)+xmax cos(bn-b)

[0025] yt=yn+ymax cos(bn-b)+xmax sin(bn-b)

[0026] Where (xmax, ymax) are the coordinates of the maximum response peak relative to the center of its corresponding inference sub-image. Based on the obtained new position (xt, yt), the coherent filter template is updated to ht = (1-γ)ht-1 + γht', where ht represents the filter template generated at time t, ht-1 is the filter template generated at time t-1, ht' is the filter template constructed based on the feature extraction results of the target's location at time t, and γ is a set constant. The relative rotation angle is updated. (bn-b);

[0027] S6: Determine whether the tracing task has been completed. If it has been completed, end the program. If the tracing task has not been completed, repeat step S4.

[0028] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0029] 1) Leveraging the powerful advantages of the fiber optic image bundle type bionic compound eye system, such as its large field of view and excellent and uniform image quality of each sub-image, accurate tracking of moving targets can be achieved within a large field of view.

[0030] 2) It is not affected by the discontinuity of the sub-images in the bionic compound eye system and the irregular angular rotation of the sub-images. Attached Figure Description

[0031] Figure 1 This is a flowchart of a target tracking method applicable to a fiber optic bundle-type bionic compound eye according to the present invention;

[0032] Figure 2 A schematic diagram of target tracking based on a fiber optic image beam-type bionic compound eye. ;

[0033] Figure 3 The following are the results of preprocessing the compound eye image according to an embodiment of the present invention: (3a) the result of obtaining the center of each sub-image; (3b) the pattern used when obtaining the rotation angle of each sub-image; (3c) the target pattern acquired using the fiber optic image bundle type bionic compound eye; and (3d) the result of image processing on the acquired image.

[0034] Figure 4 This is an example diagram showing the sequence number of each sub-image and the center coordinates of some sub-images in an embodiment of the present invention;

[0035] Figure 5 The image shows the target recognition result of the fiber optic image bundle type bionic compound eye system according to an embodiment of the present invention; (5a) is a binary moving target sub-image, and (5b) is the moving target recognition result;

[0036] Figure 6 This is a diagram showing the target tracking results when a moving target moves within a single sub-image.

[0037] Figure 7 A schematic diagram illustrating the method for determining related sub-images;

[0038] Figure 8 This is a diagram showing the target tracking results when the target moves between different sub-images.

[0039] Figure label:

[0040] 1. Target recognition results. Detailed Implementation

[0041] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0042] like Figure 1 As shown, the target tracking method for a fiber optic image bundle-type bionic compound eye according to the present invention mainly consists of two parts: a preprocessing part for the bionic compound eye image and a target recognition and tracking part. The specific implementation steps of the method are as follows:

[0043] Preprocessing of bionic compound eye images:

[0044] S1. Perform bionic compound eye image preprocessing to obtain the center coordinates and rotation angles of each sub-image: First, acquire the white wall image using the compound eye system. Based on the imaging characteristics of the fiber optic image bundle type bionic compound eye, obtain a bionic compound eye image including multiple circular sub-images. The specific results of the bionic compound eye image are as follows: Figure 3 Figure (3a) shows the result image of the center of each sub-image; then, the Hough transform algorithm is used to detect each sub-image to obtain the center coordinates and radius of each sub-image; the design is as follows Figure 3 The pattern used for rotating the sub-images shown in (3b) is printed on A4 paper. The printed image is acquired using a compound eye system to obtain the moving target image. During the acquisition process, it is necessary to ensure that the entire pattern is completely imaged within a single sub-image. Then, the target sub-image is binarized and pixel values ​​are flipped to remove the influence of noise and dust. Connected component operations are used to extract the connected component where the binarized moving target image is located: the centroid algorithm is used to obtain the centroid coordinates of the entire connected component, and the pixel values ​​of all pixels within the connected component with the centroid as the center and r as the radius are set to zero, resulting in the following: Figure 3 The binarized moving target image shown in Figure 3d is used. The centroid algorithm is employed to calculate the centroid coordinates of two connected components in the binarized moving target image. Based on these centroid coordinates, the slope of the straight line passing through these two points is calculated. Finally, the rotation angle of the sub-image is deduced from the slope of the straight line. Table 1 shows the center coordinates and rotation angles of some moving target images, where positive values ​​represent counterclockwise rotation and negative values ​​represent clockwise rotation.

[0045] Table 1

[0046]

[0047] Target identification and tracking section:

[0048] S2, the first three frames of the video sequence are processed using a three-frame interpolation algorithm to obtain, as shown below. Figure 5The three-frame difference algorithm processing result shown in Figure (5a), i.e., the binarized moving target image, is processed by connected component processing to obtain the center coordinates (xt0, yt0) of the connected component corresponding to the moving target and the size of the bounding rectangle. To verify the accuracy of the target recognition operation, a rectangle is drawn on the initial frame image based on the obtained target center coordinates (xt0, yt0) and the size of the bounding rectangle. The moving target recognition result is as follows. Figure 5 As shown in (5b), determining the moving target includes obtaining the initial pixel position (xt0, yt0) of the moving target and other basic information, including the center coordinates (x0, y0) of the sub-image where the moving target is located, the rotation angle b, and the relative rotation angle. The relative rotation angle Initialize to 0;

[0049] S3, taking the initial pixel position of the moving target as the center, extracts the image features of the moving target in a region with an area 5 times the size of the two-dimensional projection area of ​​the moving target, and generates a coherent filter template for the moving target.

[0050] S4, construct the rotation matrix, as shown below:

[0051] The rotation matrix is ​​used to rotate the next frame image. Features are extracted from the new target region of the next frame image, and then convolved with a coherent filter template. The new target region is defined by... Centered on the target, the area is five times the size of the two-dimensional projected area of ​​the moving target.

[0052] Since the target area corresponding to the moving target may change after rotation, the position of the moving target in the previous frame image is used. And the rotation matrix, to obtain the result of rotation and coordinates Corresponding pixel position coordinates Furthermore, after rotation, the four edge points of the image will move away from the original image boundary. Therefore, the rotated image will be automatically filled, causing a certain offset in the pixel coordinates of each point. Thus, it is necessary to calculate the coordinates (pi, qi) of the four edge points after rotation to obtain the offset of each pixel coordinate. The offset magnitude in the X-axis direction is Δx = |(pi)min|, and the offset magnitude in the Y-axis direction is Δy = |(qi)min|. Therefore, the coordinates corresponding to the moving target after rotation are...

[0053] S5: Based on the comparison between the response peak value and the set threshold value, determine whether the moving target image has left the current sub-image. Set a threshold 'a', which is determined based on the variation range of the response peak value corresponding to the moving target. The value of 'a' is generally around 0.1. If the response peak value η obtained by convolution operation using the coherent filter template is greater than 'a', it is considered that the moving target image has not left the current sub-image. If the obtained response peak value η is less than 'a', it is considered that the moving target image has left the current sub-image. Proceed sequentially:

[0054] N1, determine the 3 most relevant sub-images;

[0055] N2, construct a rotation matrix based on the rotation angle of each sub-image;

[0056] N3, feature extraction of the moving target region;

[0057] N4 is used to perform convolution operations to determine the position of the moving target;

[0058] The specific details of the relevant operations are as follows:

[0059] ① When the moving target image has not left the current sub-image, the new position of the moving target is obtained based on the position of the obtained response peak, that is, the target tracking result when the moving target moves within a single sub-image is obtained, such as... Figure 6 As shown. ② When the moving target image has left the current sub-image, determine the three most relevant sub-images based on the moving target's direction of motion within the current sub-image and the distance relationships between the current sub-image and other sub-images. For example... Figure 7 As shown, if the trajectory of a moving target in the current sub-image is from point O to point A, taking into account the rotation of this sub-image, the direction of movement of the moving target in the image should be the OB direction. Furthermore, the image formed by the compound eye system is inverted, so in reality, the trajectory of the moving target is closer to OC. Considering both the distance between other sub-images and the current sub-image, and the angle between the line connecting the centers of other sub-images and the current sub-image and OC, we can obtain three inference sub-images indicating the most likely location of the moving target, as shown below. Figure 7 As shown by the dashed line. Extracting features from the next frame of the image within the new target region and convolving them with a coherent filter template can prevent target tracking failure due to errors in previous judgments.

[0060] Obtain the center coordinates (xn, yn) and rotation angles b1, b2, and b3 of the three inference sub-images. Based on their respective center coordinates and rotation angles (b1-b), (b2-b), and (b3-b) relative to the initial sub-image, construct the following rotation matrix:

[0061]

[0062] Where θn is the rotation angle of the three inference sub-images relative to the initial sub-image, and n is the number of the three inference sub-images. Based on their respective rotation matrices, the image of the current frame is rotated, and the coordinates (x'n, y'n) of the center of each inference sub-image after rotation, and the coordinates (pni, qni) of the four edge points of the compound eye image after rotation are obtained using the rotation matrices. (where n is the number of the three inference sub-images, and i is the number of the four edge points; because the rotation matrix used for feature extraction of each inference sub-image is different, the corresponding points of the four edge points are different after each rotation). Feature extraction is performed sequentially on the region centered at (x'n + Δxn, y'n + Δyn) and with the same area size as the initial sub-image. The minimum horizontal coordinate value of the four edge points of the compound eye image is Δxn = |(pni)min|, and the minimum vertical coordinate value of the four edge points of the compound eye image is Δyn = |(qni)min|. A convolution operation is performed using the coherent filter template and the extracted features, f(z) = h*z, where f(z) is the obtained response result, h is the generated coherent filter template, and z is the extracted feature result in the prediction region. The position of the maximum response peak and the position of the maximum response peak value are recorded, as well as the rotation angle (bn-b) of the corresponding sub-image relative to the initial sub-image, and the center coordinates (x'n, y'n). The relative rotation angle is then updated. Given (bn-b), the latest position of the moving target is obtained according to the following formula:

[0063]

[0064] Where (xmax, ymax) are the coordinates of the center of the inference sub-image corresponding to the maximum response peak, and (xn, yn) are the center coordinates of the corresponding inference sub-image. Based on the obtained new position (xt, yt), the coherent filter template is updated to ht = (1-γ)ht-1 + γht', where ht represents the filter template generated at time t, ht-1 is the filter template generated at time t-1, ht' is the filter template constructed based on the feature extraction results of the target's location at time t, and γ is a set constant. The relative rotation angle is updated. (bn-b);

[0065] S6. Finally, determine whether the tracing task has been completed. If the tracing task has been completed, end the program. If the tracing task has not been completed, continue to process the next image according to the flowchart.

[0066] The feature extraction, generation of coherent filter templates, and convolution operations in the above process are all based on the kernel coherent filter algorithm (KCF).

[0067] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principle of the present invention should also be considered to fall within the scope of protection of the present invention.

Claims

1. A target tracking method suitable for fiber optic image bundle type bionic compound eye, characterized in that, Includes the following steps: S1, preprocess the bionic compound eye image, which includes multiple sub-images, acquired by the compound eye system, to obtain the center coordinates and rotation angle of each sub-image; S2 uses a three-frame interpolation algorithm to identify moving targets and obtain the initial pixel positions of the moving targets. The basic information of the initial sub-image containing the moving target includes the center coordinates (x0, y0) and rotation angle b of the initial sub-image, and the relative rotation angle. Initialize to 0; S3, extract features in the moving target region and generate a coherent filter template for the moving target; S4, construct the rotation matrix T, as shown below: The next frame image is rotated using a rotation matrix T until the rotation angle of the current sub-image is the same as that of the initial sub-image. The position of the moving target pixel is then determined using the rotation matrix T. The coordinates of the moving target pixel position after rotation are calculated from the pixel coordinates of the four edge points of the compound eye image. and the coordinates of the four edge pixels (p i ,q i Based on the coordinates of the four edge pixels, the offset magnitude Δx in the X-axis direction and the offset magnitude Δy in the Y-axis direction after image rotation are obtained. Based on the above information, the predicted region of the moving target is obtained. Feature extraction is performed on the predicted region, and convolution operation is performed with the coherent filter template: f(z) = h*z, where f(z) is the obtained response result, h is the generated coherent filter template, and z is the feature result extracted in the predicted region. The response peak value and the response peak value position are recorded. S5, perform the following operations sequentially based on the relationship between the response peak value η and the set threshold a, to determine whether the moving target has left the current sub-image in the prediction area; If the response peak η is greater than the set threshold a, it indicates that the moving target has not left the current sub-image. The new position of the moving target is obtained based on the position of the response peak and used as the target tracking result. If the peak response η is less than the set threshold a, it is determined that the moving target has left the current sub-image. Then, based on the positional relationship between other sub-images and the current sub-image, and the direction of motion of the moving target in the current sub-image, N1, determine the three most likely inference sub-images where the moving target appears, and obtain the center coordinates (x, y) of the three inference sub-images. n ,y n And rotation angles b1, b2, b3, and thus the rotation angles θ1 = (b1 - b), θ2 = (b2 - b), θ3 = (b3 - b) relative to the initial sub-image are obtained, collectively referred to as θ n ; N2, construct the rotation matrix based on the rotation angles of the three inference sub-images relative to the initial sub-image, as shown in the following expression: Where, θ n Let x be the rotation angle of the three inference sub-images relative to the initial sub-image. n ,y n ) represents the center coordinates of the three reasoning sub-images, and n represents the number of the reasoning sub-image, n = 1, 2, 3, until the rotation angle of the three reasoning sub-images is consistent with that of the initial sub-image, thus obtaining the reasoning region of the moving target; N3, extract features from the inference region and perform convolution operation with the coherent filter template to obtain the response peaks of the three inference regions and the positions of the response peaks of the three inference regions, and take the maximum response peak. N4 is used to perform convolution operations to determine the new position of the moving target: x t =x n -y max sin(b n -b)+x max cos(b n -b) and t / and n +and max cos(b n -b)+x max sin(b n -b) Among them, (x max ,y max ) represents the coordinate position of the maximum response peak relative to the center of the three inference sub-images, based on the obtained new position (x t ,y t Update the coherent filter template to h t =(1-γ)h t-1 +γh t ',h t h represents the filter template generated at time t. t-1 h is the filter template generated at time t-1. t 'This is a filter template constructed based on the feature extraction results of the target's location at time t, where γ is a set constant, and the relative rotation angle is updated.' For (b) n -b); S6: Determine whether the tracing task has been completed. If it has been completed, end the program. If the tracing task has not been completed, repeat step S4.

2. The target tracking method for a fiber optic image bundle type bionic compound eye according to claim 1, characterized in that, In step S3, the moving target region is a region centered on the initial pixel position of the moving target, with an area five times the size of the two-dimensional projected area of ​​the moving target.

3. The target tracking method for a fiber optic image bundle type bionic compound eye according to claim 1, characterized in that, In step S4, the pixel position coordinates of the moving target after rotation are obtained through the rotation matrix. The pixel coordinates of the four edge points of the compound eye image containing the moving target after rotation (p i ,q i ).

4. The target tracking method for a fiber optic image bundle type bionic compound eye according to claim 1, characterized in that, In step S4, the prediction region is... Centered on a region whose area is five times the size of the two-dimensional projected area of ​​the moving target.

5. A target tracking method for a fiber optic image bundle type bionic compound eye according to claim 1, characterized in that, Use the position of the moving target in the previous frame. And the rotation matrix, to obtain the result of rotation and coordinates Corresponding position coordinates After rotation, some parts of the image are automatically filled, causing the pixel coordinates of each point to shift. The coordinates (p) of the four edge points of the image after rotation are calculated. i ,q i This is used to obtain the offset of each pixel coordinate, including the offset magnitude Δx in the X-axis direction = |(p i ) min The magnitude of the offset in the Y-axis direction is Δy = |(q) i ) min | 6. The target tracking method for a fiber optic image bundle type bionic compound eye according to claim 1, characterized in that, The threshold 'a' is determined based on the range of variation of the peak response corresponding to the moving target.

Images