A binocular camera human target tracking and positioning method based on an improved Yolov7 model

Abstract

The application relates to a binocular camera human target tracking and positioning method based on an improved Yolov7 model, and comprises the following steps: 1, two same-specification cameras in a binocular vision camera are installed at the highest point covering a power construction site, and the two cameras are installed on the same installation plane; 2, image acquisition is carried out to obtain a left image and a right image; 3, the left image is directly sent into a trained improved Yolov7 detection model to obtain target ID and human region position information; 4, a normalized cross-correlation method is used to find the most similar region center point as a target tracking trajectory point; 5, the human target region center point is anchored on the left image by the Yolov7 model; 6, a weighted support window region matching cost method is used for pixel point matching on the right image; 7, disparity calculation is carried out; 8, the spatial position of the human target tracking region center is determined to realize target spatial positioning and tracking. The application can improve tracking efficiency and realize high-precision human target spatial position positioning.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a binocular camera method for human target tracking and localization based on an improved Yolov7 model. Background Technology

[0002] With the continuous development of many emerging technologies such as computer vision and artificial intelligence in recent years, target recognition and localization, as a comprehensive technology involving knowledge from multiple disciplines, have also been promoted and developed like never before. Various target detection methods have been developed, such as deep learning models like Faster R-CNN, Master R-CNN, and YOLO. Among them, the YOLO series of models have high efficiency, flexibility, and good generalization performance, and therefore have been widely used.

[0003] In target detection, monocular images can only show the target's position in the image, not its spatial location, making it difficult to maximize their value in applications. Binocular vision, an extension of monocular vision, uses the principle of parallax and imaging devices to acquire two images of the object from different positions. By calculating the positional deviation between corresponding points in the images, it obtains the object's three-dimensional geometric information. Researchers both domestically and internationally have conducted in-depth research and developed a series of stereo matching algorithms, which can be divided into three main categories: local matching algorithms, global matching algorithms, and semi-local matching algorithms. Among these, local and semi-local algorithms are the most widely used, such as BM and SGBM algorithms. Typically, stereo matching algorithms require finding corresponding matching points in the right image for points in the left image. The difference in the horizontal coordinates of these matching points is called parallax, which determines the depth information of the target. However, due to sensor noise, large areas of no texture, weak texture, or repetitive texture in the images, as well as differences in lighting received by the left and right cameras, the matching points found in the right image may fail or be incorrectly matched. This directly leads to larger depth calculation errors, resulting in low reliability of target spatial positioning and hindering its widespread application.

[0004] In the process of safety management at power grid engineering work sites, conventional technologies are currently unable to achieve high-precision positioning of personnel at the work site, resulting in situations where safety risks cannot be automatically identified. CN114494462A discloses a binocular camera ranging method based on YOLOv5 and an improved tracking algorithm, which improves the YOLOv5 model by incorporating an attention mechanism. This patent is for binocular ranging, and during the disparity calculation process, mismatches or matching failures occur when matching pixels on the left image with pixels on the right image, leading to increased distance deviation.

[0005] While the YOLOv7 model exhibits excellent detection efficiency for common categories, it lacks strong detection performance in specialized scenarios such as power grids, necessitating improvements in feature representation capabilities. Furthermore, in existing YOLO-based tracking processes, directly using YOLO detection bounding boxes easily leads to target loss due to low detection accuracy. Therefore, there is an urgent need for a stable and practical human target tracking and localization method that considers both the actual safety needs of power distribution network operations and the safety requirements of workers. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a highly efficient and accurate method for human target tracking and localization using a binocular camera based on an improved Yolov7 model.

[0007] To address the aforementioned problems, this invention provides a binocular camera-based human target tracking and localization method based on an improved Yolov7 model, comprising the following steps:

[0008] (1) Install two identical cameras from the binocular vision camera at the highest point covering the power construction site, and ensure that the two cameras are installed on the same mounting plane;

[0009] (2) Acquire images from the calibrated binocular vision camera to obtain the left image captured by the left camera and the right image captured by the right camera;

[0010] (3) The left image is directly fed into the trained improved Yolov7 detection model, and the target ID and human body region location information are obtained after detection.

[0011] (4) Using step (3) to detect human targets in the previous and current frames, first locate the approximate position of the current target in the image through the trajectory prediction model, then divide the human target area in the current frame into multiple regions, and then use the normalized cross-correlation method to find the center point of the most similar region as the target tracking trajectory point.

[0012] (5) The center point of the human target region anchored by the Yolov7 model in the left image;

[0013] (6) A weighted support window region matching cost method is used on the right image to perform pixel matching on the center points of the regions obtained from the left image;

[0014] (7) Calculate disparity using the principle of binocular vision;

[0015] (8) Based on the principle of binocular ranging, the spatial position of the center of the human target tracking area is determined, thereby realizing target spatial positioning and tracking.

[0016] The improved Yolov7 detection model trained in step (3) refers to training the sample set using the Yolov7 model and employing the DenseNet backbone feature extraction network, so that the obtained training data only contains human targets.

[0017] In step (4), the trajectory prediction model uses the Kalman filter method to predict the new target position information in the current frame based on the target region of the previous frame.

[0018] The weighted support window region matching cost method in step (6) refers to: letting the disparity d of pixel p in the left image be... p

[0019]

[0020] In the formula: p x It is the x-coordinate of pixel p; p y It is the y-coordinate of pixel p; a fp ,b fp and c fp It is the parallax plane f p The parameters, where the disparity plane satisfies

[0021]

[0022] Where: Γ is the set of disparity planes; m(p,f) represents the aggregation cost, expressed as...

[0023]

[0024] In the formula: W p It is the supporting window for the position of point p in the left image;

[0025] ω(p,q) is the weight that measures the matching between two pixels p and q, and it is described by color.

[0026]

[0027] In the formula, color(p,q) is the color distance between pixels p and q; γ c Indicates the scale parameter;

[0028] The cost calculation function ρ(q,q') is

[0029]

[0030] In the formula: I q ,I q’ These represent the gray levels of points q and q', respectively. and τ represents the gradient values ​​at points q and q'; c τ is the grayscale threshold;g This is the gradient threshold.

[0031] Compared with the prior art, the present invention has the following advantages:

[0032] 1. This invention uses the superior YOLOv7 model and incorporates the DenseNet module to enhance the feature transmission capability of the backbone feature extraction network and improve detection performance.

[0033] 2. This invention employs Kalman filtering prediction and introduces a local region matching method to improve tracking efficiency during the tracking process.

[0034] 3. To avoid the problem of poor positioning accuracy caused by the mismatch between the center point of the target position in the left image and the right image, this invention utilizes the stereo vision principle of binocular cameras and adopts a parallax calculation method based on local area matching to calculate the spatial position of the target point, thereby achieving high-precision spatial positioning of human targets. Attached Figure Description

[0035] Figure 1 This is a flowchart of the present invention.

[0036] Figure 2 This is the backbone feature extraction network of the Yolov7 model in this invention.

[0037] Figure 3 This is a schematic diagram of the binocular vision ranging principle of the present invention. Detailed Implementation

[0038] During tracking, interference can easily occur due to factors such as target occlusion, and lighting conditions can also affect tracking and localization. To avoid this, this invention divides the human target region obtained from two adjacent frames detected by YOLOv7 into multiple regions and uses a normalized cross-correlation method to find the center of the most similar region as the target for tracking. Furthermore, based on the principle of binocular vision ranging, a local region matching method is provided to calculate the spatial position of the target point, thereby achieving human target tracking and localization.

[0039] The binocular camera or binocular vision described in this invention uses two cameras on the same plane to capture scene images, providing a guarantee for subsequent calculation of target distance and spatial position.

[0040] In this invention, Kalman filtering is used to predict the target location in the next frame of the image.

[0041] In this invention, target matching means that the target regions selected in two frames of images have a certain similarity, and therefore the target regions are considered to be matched.

[0042] In this invention, target tracking involves extracting the motion trajectory of the target in a video.

[0043] like Figure 1 As shown, a binocular camera method for human target tracking and localization based on an improved Yolov7 model includes the following steps:

[0044] (1) Install two identical cameras from the binocular vision camera at the highest point covering the power construction site, and ensure that the two cameras are installed on the same mounting plane.

[0045] (2) Acquire images from the calibrated binocular vision camera to obtain the left image captured by the left camera and the right image captured by the right camera.

[0046] (3) The left image is directly fed into the trained improved Yolov7 detection model, and the target ID and human body region location information are obtained after detection.

[0047] Among them, the well-trained improved Yolov7 detection model refers to the sample set being trained using the Yolov7 model and the DenseNet backbone feature extraction network, and the obtained training data only contains human targets.

[0048] To minimize the impact of lighting, weather, and other interference in real-world images, the Yolov7 model employs a DenseNet backbone feature extraction network, such as... Figure 2 As shown. In Figure 2 In this model, DenseNet is added before the prediction model to obtain multi-layer convolutional features using two scales, 13×13 and 26×26, thereby enhancing the feature transfer capability of the backbone feature extraction network and improving the model's detection performance.

[0049] (4) Using step (3) to detect human targets in the previous and current frames, first, the approximate location of the current target in the image is located through a trajectory prediction model. Then, the human target region in the current frame is divided into multiple regions, and the center point of the most similar region is found as the target tracking trajectory point using a normalized cross-correlation method. The specific process is as follows:

[0050] The trajectory prediction model uses the Kalman filter method to predict the new target position information in the current frame based on the target region in the previous frame.

[0051] To prevent segmentation failure due to inconsistent region sizes, the human target region in the current frame and the target region in the previous frame are adaptively scaled to the same size and uniformly divided into N regions.

[0052] N=(2k+1)×(2k+1) (1)

[0053] In the formula: k is a positive integer.

[0054] For each region, calculate the normalized cross-correlation coefficient.

[0055]

[0056] In the formula: (u,v) is the center position of each region; I1 and I2 correspond to the local regions of the human target in the previous frame and the current frame, respectively; μ1 and μ2 are the mean values ​​of each region.

[0057] To avoid multiple duplicate matching regions, a matching method that merges adjacent regions to calculate the NCC coefficient is added. Simultaneously, the Hungarian algorithm is used to obtain matching regions based on the correspondence of region numbers, thereby reducing tracking errors in cases of target aliasing. The region that best matches the current frame with the previous frame is the current target position, and the center point of the region is the target tracking trajectory point.

[0058] In local region matching, probability density similarity calculation methods can be used to match targets.

[0059] (5) The center point of the human target region anchored by the Yolov7 model on the left image.

[0060] ⑹ To avoid the inability to find matching points on the right image due to factors such as lighting and image texture, a weighted support window region matching cost method is used on the right image to perform pixel matching on the center points of the regions obtained from the left image.

[0061] Wherein: the weighted support window region matching cost method refers to: letting the disparity d of pixel p in the left image be... p

[0062]

[0063] In the formula: p x It is the x-coordinate of pixel p (on the image); p y It is the y-coordinate of pixel p (on the image); a fp ,b fp and c fp It is the parallax plane f p The parameters (i.e., weights) of the disparity plane satisfy the following:

[0064]

[0065] Where: Γ is the set of disparity planes; m(p,f) represents the aggregation cost, expressed as...

[0066]

[0067] In the formula: W p It is the supporting window for the position of point p in the left image;

[0068] ω(p,q) is the weight that measures the matching between two pixels p and q, and it is described by color.

[0069]

[0070] In the formula: color(p,q) is the color distance between pixels p and q; γ c Indicates the scale parameter;

[0071] The cost calculation function ρ(q,q') is

[0072]

[0073] In the formula: I q ,I q’ These represent the gray levels of points q and q', respectively. and τ represents the gradient values ​​at points q and q'; c τ is the grayscale threshold; g Set the gradient threshold; set the grayscale threshold τ. c and gradient threshold τ g This is to avoid the cost function value deviating from reality due to excessively large values ​​during the calculation process.

[0074] The cost is calculated using a weighted method to find the optimal aggregation cost m(p,f).

[0075] (7) Calculate disparity using the principle of binocular vision.

[0076] When using binocular vision to calculate spatial position, a method for calculating disparity using a disparity plane is employed. This method uses unit normal vectors and coordinate point information to measure the disparity plane, enabling the disparity values ​​of matching points to be obtained on both the left and right disparity maps. This avoids larger errors caused by not finding matching points or mismatches on the right image.

[0077] parallax plane f p It can be derived from point (x) p ,y p ,z p ) and unit normal vector (n x ,n y ,n z The parallax plane parameters are determined by the normal vector corresponding to the pixel position.

[0078]

[0079] In the formula, p is a point on the parallax plane, and x p ,y p Let x and y be the coordinates of point p, and z be the coordinates of point p. pThis represents the disparity value. The unit normal vector is obtained iteratively through the normal vectors of the matching pixel points within the support window. The basic iterative process is as follows:

[0080] ① Randomize the normal vector and normalize it to obtain (n x ,n y ,n z );

[0081] ② Iteratively search for the minimum aggregation cost m(f,p) within the support window using equation (8), and record the unit normal vector at that point;

[0082] ③ Repeat steps ① and ②, update the unit normal vector corresponding to the minimum aggregation cost compared to the previous one, and calculate the disparity value d using this unit normal vector. p .

[0083] Depend on Figure 3 As can be seen, the pixels corresponding to objects in the left and right images are different in the horizontal direction. Therefore, it is necessary to match this point. The difference in pixel position in the x direction is called parallax. After matching, the distance can be calculated by using the camera's intrinsic and extrinsic parameters and isosceles triangles.

[0084] (8) Based on the principle of binocular ranging, the spatial position of the center of the human target tracking area is determined, thereby achieving target spatial localization and tracking. The specific process is as follows:

[0085] For binocular ranging, such as Figure 3 The left and right cameras shown are of the same model and are mounted on the same horizontal line. A method for calculating their spatial positions is constructed in the world coordinate system.

[0086]

[0087] Where: z is the distance between the camera and the target point; x l x is the pixel distance from the center of the left image; r x is the pixel distance from the center of the image on the right; l -x r The result obtained is the disparity value d. p ;f s 'b' represents the focal length of the binocular camera; 'b' represents the baseline distance between the binocular cameras.

Claims

1. A method for human target tracking and localization using a binocular camera based on an improved Yolov7 model, comprising the following steps: (1) Install two identical cameras from the binocular vision camera at the highest point covering the power construction site, and ensure that the two cameras are installed on the same mounting plane; (2) Acquire images from the calibrated binocular vision camera to obtain the left image captured by the left camera and the right image captured by the right camera; (3) The left image is directly fed into the trained improved Yolov7 detection model, and the target ID and human body region location information are obtained after detection. (4) Using step (3) to detect human targets in the previous and current frames, first locate the approximate position of the current target in the image through the trajectory prediction model, then divide the human target area in the current frame into multiple regions, and then use the normalized cross-correlation method to find the center point of the most similar region as the target tracking trajectory point. (5) The center point of the human target region anchored by the Yolov7 model in the left image; (6) A weighted support window region matching cost method is used on the right image to perform pixel matching on the center points of the regions obtained from the left image; the weighted support window region matching cost method refers to: letting the disparity d of pixel p on the left image be... p In the formula: p x It is the x-coordinate of pixel p; p y It is the y-coordinate of pixel p; a fp ,b fp and c fp It is the parallax plane f p The parameters, where the disparity plane satisfies Where: Γ is the set of disparity planes; m(p,f) represents the aggregation cost, expressed as... In the formula: W p It is the supporting window for the position of point p in the left image; ω(p,q) is the weight that measures the matching between two pixels p and q, and it is described by color. In the formula, color(p,q) is the color distance between pixels p and q; γ c Indicates the scale parameter; The cost calculation function ρ(q,q′) is In the formula: I q ,I q’ These represent the gray levels of points q and q', respectively. and τ represents the gradient values ​​at points q and q'; c τ is the grayscale threshold; g Gradient threshold; (7) Calculate disparity using the principle of binocular vision; (8) Based on the principle of binocular ranging, the spatial position of the center of the human target tracking area is determined, thereby realizing target spatial positioning and tracking.

2. The method for human target tracking and localization using a binocular camera based on an improved Yolov7 model as described in claim 1, characterized in that: The improved Yolov7 detection model trained in step (3) refers to training the sample set using the Yolov7 model and employing the DenseNet backbone feature extraction network, so that the obtained training data only contains human targets.

3. The method for human target tracking and localization using a binocular camera based on an improved Yolov7 model as described in claim 1, characterized in that: In step (4), the trajectory prediction model uses the Kalman filter method to predict the new target position information in the current frame based on the target region of the previous frame.

Images