Object tracking method and device based on binocular camera
Through the binocular camera-based object tracking method, the problem that shooting tools in the prior art cannot automatically track and focus are solved, automatic tracking and automatic focus are realized, shooting quality is improved, and 24-hour capture effect is ensured.
Patent Information
- Application Number
- CN202411275642.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-12
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2044-09-12
AI Technical Summary
Existing shooting tools cannot automatically track and focus, making it difficult to guarantee the quality of children's photos or videos, and cannot achieve 24-hour capture, and many exciting moments may be missed.
Using the object tracking method based on binocular camera, by obtaining multiple sets of shot images captured by binocular camera, feature extraction and voxel set construction are performed, matching costs are calculated to determine the region of interest, and the movement direction of the moving module is adjusted to make The target object is located in the center of the binocular camera screen, and is automatically focused through cross-order Fourier transform.
It realizes automatic tracking and automatic focus, improves shooting quality, has the advantage of anti-occlusion, and can capture target objects 24 hours a day, ensuring that wonderful moments are not missed.
Smart Images

Figure CN118799819B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of object tracking, and in particular to an object tracking method and device based on a binocular camera. Background Art
[0002] At present, the photo or video shooting of children still relies mainly on manual methods, usually by parents or elders holding a camera to shoot the children. This method is not only time-consuming and laborious, but also cannot achieve 24-hour all-weather capture, and many wonderful moments may be missed inadvertently. Although some fixed shooting tools on the market can shoot automatically, they lack functions such as automatic tracking and automatic focusing, making it difficult to guarantee the shooting quality. Summary of the invention
[0003] In order to solve the above-mentioned problems in the prior art, the present invention provides an object tracking method and device based on a binocular camera, which solves the problem in the prior art that the shooting tool cannot automatically track and automatically focus.
[0004] According to one aspect of the present invention, there is provided an object tracking method based on a binocular camera, comprising:
[0005] Acquire multiple sets of captured image sets captured by a binocular camera; wherein each set of captured image sets includes a left view and a right view;
[0006] Performing feature extraction on the left view to obtain a first image feature;
[0007] Performing feature extraction on the right view to obtain a second image feature;
[0008] constructing the first image feature and the second image feature into a voxel set;
[0009] Obtaining a matching cost for each voxel in the voxel set;
[0010] Determine a region of interest according to a matching cost and a matching condition of each voxel in the voxel set; wherein the region of interest includes a target object;
[0011] Based on the image position of the target object, the moving direction of the moving module is adjusted so that the target object is located at the center of the picture when the binocular camera is shooting; wherein the binocular camera is arranged above the moving module, and the moving module drives the binocular camera to move.
[0012] In one embodiment, obtaining the matching cost of each voxel in the voxel set includes:
[0013] Normalizing each voxel in the voxel set to obtain each normalized voxel in the voxel set;
[0014] Performing Fourier position coding processing on each normalized voxel in the voxel set to obtain each encoded voxel in the voxel set;
[0015] According to each encoded voxel in the voxel set, obtaining each fusion position code in the voxel set;
[0016] According to each fused position code in the voxel set, a matching cost of each voxel in the voxel set is calculated.
[0017] In one embodiment, obtaining each fusion position code in the voxel set according to each encoded voxel in the voxel set includes:
[0018] determining a centroid of a body mass in the set of voxels;
[0019] Determine a projection value of the centroid of the body in the left view;
[0020] Determine a projection value of the centroid of the body in the right view;
[0021] Based on a multi-layer perceptron, each fusion position code in the voxel set is generated according to the projection value of the body mass centroid in the left view, the projection value of the body mass centroid in the right view, and each encoded voxel in the voxel set.
[0022] In one embodiment, calculating the matching cost of each voxel in the voxel set according to each fused position code in the voxel set includes:
[0023] Based on the sampling condition, sampling processing is performed on the first image feature to obtain a sampled first image feature;
[0024] Based on the sampling condition, sampling processing is performed on the second image feature to obtain a sampled second image feature;
[0025] Perform feature splicing processing on each fusion position code in the voxel set, the sampled first image feature, and the sampled second image feature to obtain a spliced feature;
[0026] Based on a multi-layer perceptron, a matching cost of each voxel in the voxel set is calculated according to the concatenated features.
[0027] In one embodiment, the sampling process of the first image feature based on the sampling condition to obtain the sampled first image feature includes:
[0028] The first image feature is sampled by surrounding each voxel in the voxel set with the body centroid of the voxel set at a preset step length to obtain the sampled first image feature.
[0029] In one embodiment, determining the region of interest according to the matching cost and matching condition of each voxel in the voxel set includes:
[0030] Downsampling the matching cost of each voxel in the voxel set to obtain a matching voxel;
[0031] Calculating the similarity between the matching voxels in the voxel set and the matching voxels in other voxel sets within a unit time step;
[0032] Based on multiple similarities, the region of interest is determined.
[0033] In one embodiment, determining the region of interest according to multiple similarities includes:
[0034] Selecting the plurality of matching voxels whose similarities are greater than a preset similarity threshold as target voxels;
[0035] The target voxels are grouped into regions of interest.
[0036] In one embodiment, before adjusting the moving direction of the moving module based on the image position of the target object, the object tracking method based on the binocular camera further includes:
[0037] Calculate edge values of multiple spans on the image within the region of interest;
[0038] Calculating an average of the edge values of the plurality of spans;
[0039] If the average value of the edge values of the multiple spans is less than the preset average value threshold, Laplace transform is performed on the pixel values of the image in the region of interest, and the focus of the binocular camera is adjusted based on the focusing scheme.
[0040] In one embodiment, the focusing process of adjusting the focal length of the binocular camera based on the focusing scheme includes:
[0041] Adjusting the focal length of the binocular camera and determining whether the pixel value after Laplace transformation is greater than a preset pixel threshold;
[0042] If the pixel value after the Laplace transformation is greater than the preset pixel threshold, determining the zoom direction of the adjusted focal length;
[0043] The focal length of the binocular camera is controlled to zoom in the zoom direction so that edge values of multiple spans of the image in the region of interest meet a preset standard.
[0044] According to another aspect of the present invention, there is provided an object tracking device based on a binocular camera, comprising:
[0045] The auxiliary image acquisition module is used to acquire multiple sets of captured images captured by the binocular camera; wherein each set of captured images includes a left view and a right view;
[0046] A target tracking module is used to extract features from the left view to obtain a first image feature; extract features from the right view to obtain a second image feature; construct the first image feature and the second image feature into a voxel set; obtain the matching cost of each voxel in the voxel set; determine a region of interest based on the matching cost and matching conditions of each voxel in the voxel set; wherein the region of interest includes a target object; based on the image position of the target object, adjust the moving direction of the moving module so that the target object is located at the center of the picture when the binocular camera is shooting; wherein the binocular camera is arranged above the moving module, and the moving module drives the binocular camera to move.
[0047] The beneficial effect of the present invention is embodied in that the present invention provides an object tracking method and device based on a binocular camera, including: obtaining a plurality of sets of captured image sets captured by a binocular camera, wherein each set of captured image sets includes a left view and a right view, extracting features from the left view to obtain a first image feature, extracting features from the right view to obtain a second image feature, constructing the first image feature and the second image feature into a voxel set, obtaining the matching cost of each voxel in the voxel set, determining a region of interest according to the matching cost and matching conditions of each voxel in the voxel set, wherein the region of interest includes a target object, adjusting the moving direction of a moving module based on the image position of the target object so that the target object is located at the center of the picture when the binocular camera is shooting, wherein the binocular camera is arranged above the moving module, and the moving module drives the binocular camera to move. A voxel set is generated by overlapping views of the binocular camera, and Fourier position coding is aligned and fused with pixel projection, and finally the tracking of the target is completed by calculating the matching cost and cosine similarity matching. Compared with the existing target tracking method, this method has the advantage of anti-occlusion. In addition, the present invention also evaluates the image quality through cross-order Fourier transform to guide the automatic focusing of the binocular camera, thereby improving the shooting quality of the binocular camera. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] Figure 1 The present invention provides a flow chart of an object tracking method based on a binocular camera.
[0049] Figure 2The figure is a flow chart of the method for determining a region of interest provided by the present invention.
[0050] Figure 3 This is a schematic diagram of the structure of the object tracking device based on the binocular camera provided by the present invention.
[0051] Figure 4 A schematic structural diagram of an object tracking device based on a binocular camera provided in another embodiment of the present invention.
[0052] Figure 5 is a structural diagram of an electronic device provided by an exemplary embodiment of the present invention.
[0053] Reference numerals:
[0054] 10. Electronic device; 11. Processor; 12. Memory; 13. Input device; 14. Output device. DETAILED DESCRIPTION
[0055] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0056] Embodiment 1: Figure 1 The flowchart of the object tracking method based on binocular camera provided by the present invention is as follows. Figure 1 As shown, the object tracking method based on the binocular camera includes:
[0057] S110: Acquire multiple sets of captured image sets captured by a binocular camera, wherein each set of captured image sets includes a left view and a right view.
[0058] In the embodiment of the present invention, the present invention uses an ultra-high-definition binocular camera with a pixel of 48 million to shoot the target object, so as to obtain multiple sets of captured image sets within a preset time period, wherein the preset time period can be a period from a certain moment in the past to the current moment, and each set of captured image sets includes a left view and right view .
[0059] S120: Extract features from the left view to obtain first image features.
[0060] In the embodiment of the present invention, since the image contains a large amount of pixel data, directly processing this data will be very complicated and computationally intensive. Therefore, by performing feature extraction on the left view, the image data can be converted into a smaller dimension, thereby simplifying subsequent processing and analysis. In addition, by extracting important features, the computational complexity of subsequent algorithms (such as classification, recognition, etc.) can be reduced, and the processing speed can be improved. In addition, the image may contain a lot of redundant or irrelevant information, and feature extraction can help screen out valuable information and reduce the impact of noise.
[0061] Specifically, the pre-trained target detection (detection of people) lightweight network EfficientNet is used as the backbone network for the left view The first-level feature extraction is performed on the image features, and then the first-level feature is enhanced by FPN to obtain the first multi-scale image feature (first image feature) ,in, The serial number representing features of different scales. FPN (Feature Pyramid Network) is a network architecture used for computer vision tasks, especially in tasks such as object detection and segmentation. FPN effectively utilizes multi-scale information by constructing a feature pyramid, thereby improving the model's ability to detect objects of different sizes.
[0062] S130: Extract features from the right view to obtain second image features.
[0063] In the embodiment of the present invention, since the image contains a large amount of pixel data, directly processing this data will be very complicated and computationally intensive. Therefore, by performing feature extraction on the right view, the image data can be converted into a smaller dimension, thereby simplifying subsequent processing and analysis. In addition, by extracting important features, the computational complexity of subsequent algorithms (such as classification, recognition, etc.) can be reduced, and the processing speed can be improved. In addition, the image may contain a lot of redundant or irrelevant information, and feature extraction can help screen out valuable information and reduce the impact of noise.
[0064] Specifically, the pre-trained target detection (detection of people) lightweight network EfficientNet is used as the backbone network for the right view The first-level feature extraction is performed on the image features, and then FPN is used to enhance the extraction of the first-level features to obtain the second multi-scale image features (second image features) ,in, The serial number representing the features of different scales.
[0065] S140: Construct the first image feature and the second image feature into a voxel set.
[0066] In the embodiment of the present invention, the first image feature and the second image feature are jointly constructed into a dimension The voxel set Q, where In Indicates length, Indicates width, High representation. The main reason for constructing voxel sets is that they can effectively represent and process 3D data, especially in 3D object recognition, reconstruction, and analysis tasks involving deep learning and computer vision. Voxel sets can retain geometric information in 3D space, allowing the model to understand the shape, volume, and spatial relationship of objects. And when processing images from different perspectives, converting features into voxel sets can effectively fuse information from multiple perspectives and provide a more comprehensive object representation.
[0067] S150: Obtain the matching cost of each voxel in the voxel set.
[0068] In the embodiment of the present invention, the matching cost can be used to evaluate the similarity of the voxel with the target model or other voxels. By calculating these costs, it can be determined which voxels match other voxels or target objects, thereby helping to achieve more accurate recognition and classification.
[0069] Specifically, S150 may include the following steps:
[0070] S151: normalize each voxel in the voxel set to obtain each normalized voxel in the voxel set.
[0071] In the embodiment of the present application, normalization can help reduce the influence of noise in the data, especially when processing unevenly distributed data, so that more attention can be paid to important features. Normalization can adjust the values of different voxels to the same scale, which helps to eliminate the influence caused by differences in voxel size, shape or distribution, making subsequent analysis and processing more consistent and reliable. Therefore, each voxel in the voxel set is normalized to obtain the normalized voxel in the voxel set.
[0072] S152: Perform Fourier position coding processing on each normalized voxel in the voxel set to obtain each encoded voxel in the voxel set.
[0073] In the embodiment of the present application, in order to capture richer spatial features and effectively capture the spatial position of voxels, each normalized voxel in the voxel set is subjected to Fourier position coding to obtain each encoded voxel in the voxel set. Fourier position coding can effectively capture the spatial position of the voxel and provide rich information about the relative position of the voxel in space. Through Fourier coding, the position information of the voxel is converted into a series of frequency components, which can capture richer spatial features.
[0074] S153: Obtain each fusion position code in the voxel set according to each encoded voxel in the voxel set.
[0075] S154: Calculate the matching cost of each voxel in the voxel set according to each fusion position code in the voxel set.
[0076] S160: Determine a region of interest according to the matching cost and matching condition of each voxel in the voxel set, wherein the region of interest includes the target object.
[0077] S170: Based on the image position of the target object, adjust the moving direction of the moving module so that the target object is located at the center of the picture when the binocular camera is shooting, wherein the binocular camera is arranged above the moving module, and the moving module drives the binocular camera to move.
[0078] In the embodiment of the present invention, after the region of interest is determined, the moving direction of the mobile module is adjusted so that the target object is located at the center of the picture taken by the binocular camera, thereby achieving tracking of the target object. The binocular camera is arranged above the mobile module, and the mobile module drives the binocular camera to move. The mobile module is composed of a two-axis gimbal that supports 360-degree horizontal pitch and plus or minus 20 degrees.
[0079] Specifically, the two-dimensional coordinates of the target object in the region of interest are obtained, and the two-dimensional coordinates can be the center point in the target image. The width and height of the picture taken by the binocular camera are calculated to determine the center position of the picture. According to the two-dimensional coordinates of the target object in the region of interest and the center position of the picture, the distance deviation is calculated, and the moving direction of the mobile module is adjusted according to the distance deviation. For example, if the horizontal coordinate deviation in the distance deviation is greater than 0, it means that the target object is located on the right side of the picture, and the mobile module drives the binocular camera to move to the left, otherwise, it needs to move to the right. If the longitudinal coordinate deviation in the distance deviation is greater than 0, it means that the target object is located at the bottom of the picture, and the mobile module drives the binocular camera to move upward, otherwise, it needs to move downward. In addition, the moving speed and direction of the mobile module are set according to the distance deviation. Specifically, a control algorithm (such as PID control) can be used to move smoothly to avoid too fast or jitter. According to the distance deviation, an adjustment instruction can be generated so that the adjustment model executes the adjustment instruction to drive the binocular camera to move, and the target object is located at the center of the picture when the binocular camera is shooting.
[0080] Optionally, when the target object is within preset pixels of the center position of the picture, the movement module is controlled to stop moving, wherein the preset pixels may be 10 pixels.
[0081] In addition, a circular touch screen is provided on the binocular camera, which is used to display the screen of the remote video connection.
[0082] The present invention provides an object tracking method based on a binocular camera, comprising: obtaining a plurality of sets of captured images captured by a binocular camera, wherein each set of captured images includes a left view and a right view, extracting features from the left view to obtain a first image feature, extracting features from the right view to obtain a second image feature, constructing the first image feature and the second image feature into a voxel set, obtaining the matching cost of each voxel in the voxel set, determining an area of interest according to the matching cost and matching conditions of each voxel in the voxel set, wherein the area of interest includes a target object, adjusting the moving direction of a moving module based on the image position of the target object so that the target object is located at the center of the picture when the binocular camera is shooting, wherein the binocular camera is arranged above the moving module, and the moving module drives the binocular camera to move. A voxel set is generated through overlapping views of the binocular camera, and Fourier position coding is aligned and fused with pixel projection, and finally the tracking of the target is completed by calculating the matching cost and cosine similarity matching. Compared with the existing target tracking method, this method has the advantage of anti-occlusion. In addition, the present invention also evaluates the image quality through cross-order Fourier transform to guide the automatic focusing of the binocular camera, thereby improving the shooting quality of the binocular camera.
[0083] Embodiment 2: In one embodiment, S153 can be specifically implemented as follows: determining the body mass centroid in the voxel set; determining the projection value of the body mass centroid in the left view; determining the projection value of the body mass centroid in the right view; based on a multi-layer perceptron, generating each fusion position code in the voxel set according to the projection value of the body mass centroid in the left view, the projection value of the body mass centroid in the right view, and each encoded voxel in the voxel set.
[0084] In the embodiment of the present application, the body centroid of the voxel set is first determined. The body centroid (VoxelCentroid) refers to the "center of gravity" or "geometric center" of a voxel set in three-dimensional space. Voxel is the basic unit in a three-dimensional grid and is usually used to represent volume data in three-dimensional space. The body centroid is defined as p. The body centroid p is determined in the left view. The projection value on , determine the centroid p of the body in the right view The projection value on ,in, Represents the projection matrix of the body centroid in the left view and Represents the projection matrix of the body's centroid on the right view.
[0085] Then, according to the projection value of the body mass centroid on the left view and the projection value of the body mass centroid on the right view, the total projection value of the body mass centroid on the left view and the right view is constructed, and each encoded voxel in the voxel set and the total projection value of the body mass centroid on the left view and the right view are input into the multi-layer perceptron for alignment and fusion to obtain the encoding of each fused position in the voxel set.
[0086] Specifically, the calculation formula of the total projection value of the body mass centroid on the left view and the right view is: , where p represents the physical fitness center, It represents the projection value of the centroid of the body on the left view. It represents the projection value of the centroid of the body on the right view. Represents the projection matrix of the body centroid in the left view and Represents the projection matrix of the body centroid on the right view, represents the first image feature, Second image feature: The projection result of the centroid of the body onto the image is usually a two-dimensional coordinate (x, y), which indicates the specific position of the centroid on the image plane.
[0087] Specifically, each fusion position in the voxel set encodes The calculation formula is: ; where q represents each encoded voxel in the voxel set.
[0088] Embodiment 3: In one embodiment, S154 can be specifically implemented as follows: based on the sampling condition, sampling the first image feature to obtain the sampled first image feature; based on the sampling condition, sampling the second image feature to obtain the sampled second image feature; performing feature splicing processing on each fusion position encoding, the sampled first image feature, and the sampled second image feature in the voxel set to obtain the spliced feature; based on the multi-layer perceptron, calculating the matching cost of each voxel in the voxel set according to the spliced feature.
[0089] In an embodiment of the present application, in order to track the target object, it is necessary to calculate the matching cost of each voxel in the voxel set. Specifically, based on the sampling condition, the first image feature and the second image feature are sampled and processed respectively to obtain the sampled first image feature and the second image feature. The sampling condition includes: sampling the first image feature around the body mass centroid of the voxel set for each voxel in the voxel set and using a preset step size to obtain the sampled first image feature. Sampling the second image feature around the body mass centroid of the voxel set for each voxel in the voxel set and using a preset step size to obtain the sampled second image feature. The preset step size can be defined as .
[0090] Then, each fusion position code, the first image feature after sampling, and the second image feature after sampling in the voxel set are subjected to feature splicing to obtain the spliced features. Since the voxel fusion position code, the first image feature, and the second image feature provide different information respectively, the voxel code contains spatial structure and volume information, and the image feature contains surface texture and color information, so through splicing, these information can be combined together to improve the expressiveness of the features. Voxel codes are usually defined in three-dimensional space, while image features are extracted on a two-dimensional plane. By splicing these features together, the spatial consistency between different data sources can be maintained, which helps the model better understand the position and shape of objects in space. And a single feature is often not enough to fully describe complex scenes or objects. By splicing multiple features, a richer feature representation can be obtained. Finally, the spliced features are input into a multi-layer perceptron to calculate the matching cost of each voxel in the voxel set.
[0091] Specifically, the calculation formula of the matching cost is:
[0092] ;in, Represents the preset step size, represents the encoding of each fusion position in the voxel set, p represents the centroid of the body, It represents the projection value of the centroid of the body on the left view. It represents the projection value of the centroid of the body on the right view. Represents the projection matrix of the body centroid in the left view and Represents the projection matrix of the body centroid on the right view, represents the first image feature, The second image feature, MLP stands for multi-layer perceptron, Indicates splicing.
[0093] Embodiment 4: In one embodiment, S154 may be specifically implemented as follows: sampling the first image feature of each voxel in the voxel set around the body centroid of the voxel set with a preset step size to obtain the sampled first image feature.
[0094] In an embodiment of the present invention, by sampling around the centroid, local features around the voxel can be captured. This is very important for understanding the context and surroundings of the voxel, especially when dealing with complex shapes or textures. Sampling centered on the centroid ensures that spatial consistency is maintained between adjacent voxels. This method helps to unify feature representations in multi-view or multi-modal data, making subsequent processing more efficient. Sampling in a local area can reduce errors caused by noise at a single sampling point. This strategy can more accurately reflect the true characteristics of the region by aggregating information from multiple sampling points. By flexibly adjusting the step size , so as to perform appropriate feature extraction in different application scenarios. For example, for areas with rich details, a smaller step size can be selected for dense sampling, while for flat areas, a larger step size can be selected for sampling.
[0095] Embodiment 5: Figure 2 FIG. 1 is a flow chart of the method for determining a region of interest provided by the present invention. Figure 2 As shown, S160 may include:
[0096] S161: Down-sample the matching cost of each voxel in the voxel set to obtain a matching voxel.
[0097] In an embodiment of the present invention, in order to reduce the impact of noise and outliers on the matching cost, thereby improving the robustness of the matching algorithm, the matching cost of each voxel in the voxel set is downsampled to obtain a matching voxel, wherein downsampling refers to reducing the resolution of data by reducing the number of samples or data points in the data set in data processing, so as to reduce the computational burden, save storage space, increase processing speed or simplify analysis.
[0098] The present invention can reduce the impact of noise and outliers on the matching cost by downsampling, thereby improving the robustness of the matching algorithm and making it more adaptable to changes and uncertainties in the data. And after downsampling, attention can be focused on more important voxels or features, which may lead to more accurate matching results. By focusing only on voxels with significant matching costs, the algorithm can more effectively find the best match. The amount of data after downsampling is small, and subsequent processing (such as optimization, fusion or classification) will be more efficient, which can speed up the entire workflow.
[0099] S162: Calculate the similarity between the matching voxels in the voxel set within the unit time step and the matching voxels in other voxel sets.
[0100] In the embodiment of the present invention, in order to determine which image regions in the multiple sets of captured images are most relevant, it is necessary to calculate the similarity between the matching voxels in the voxel set within the unit time step and the matching voxels in other voxel sets. By analyzing the similarity, important features can be extracted from the voxel set to help distinguish the region of interest from the background. By calculating the similarity, the quality of the extracted ROI (region of interest) can be evaluated to ensure that the selected area is representative and consistent.
[0101] The unit time step refers to the division of time into time intervals (steps) of fixed length in a dynamic system or simulation process. The unit time step of this application can be 1 minute or 30 seconds.
[0102] S163: Determine a region of interest based on multiple similarities.
[0103] Embodiment 6: In one embodiment, S163 may be specifically implemented as: selecting a plurality of matching voxels with similarities greater than a preset similarity threshold as target voxels; and combining the target voxels into a region of interest.
[0104] In an embodiment of the present invention, a region of interest is formed by selecting relatively similar matching voxels from multiple voxel sets to form one or more connected regions. This can be achieved by connected component analysis or region growing algorithm. According to application requirements, the shape and size of the ROI can be further optimized, for example, by using morphological operations to remove small noise areas, wherein the region of interest includes the target object.
[0105] Example 7: In one embodiment, before S170, the object tracking method based on the binocular camera can be specifically implemented as follows: calculating edge values of multiple spans on the image within the region of interest; calculating the average value of the edge values of the multiple spans; if the average value of the edge values of the multiple spans is less than a preset average value threshold, performing Laplace transform on the pixel values of the image within the region of interest and focusing the focal length of the binocular camera based on the focusing scheme.
[0106] In the embodiment of the present application, since the region of interest is the most important part of the entire image, computing resources and time can be saved by analyzing only the region of interest.
[0107] Specifically, edge values of multiple spans of the image within the region of interest are calculated, wherein the calculation formula for edge values of multiple spans on the image within the region of interest is: ;in, represents the two-dimensional p-order Fourier transform, Represents a preset order span sequence, where the preset order span sequence includes multiple spans on the image, and the spans are the range, coverage area or distribution of certain features or objects in the image. For example, multiple spans can be the spatial span of the image, that is, the size of the object or feature in the image in space. For example, the span of an object can refer to the pixel width and height it occupies in the image. Then calculate the average value of the edge values of multiple spans. If the average value of the edge values of multiple spans is less than the preset average value threshold, it means that the focal length of the binocular camera is not good. Therefore, the pixel value of the image in the area of interest is Laplace transformed, where the calculation formula of Laplace transform is:
[0108] ;in, Represents the image pixel coordinates in the region of interest.
[0109] Finally, based on the focusing scheme, the focal length of the binocular camera is adjusted so that the captured image is clearer when tracking the target object.
[0110] Example 8: In one embodiment, the object tracking method based on the binocular camera can be specifically implemented as follows: adjust the focal length of the binocular camera and determine whether the pixel value after Laplace transformation is greater than a preset pixel threshold; if the pixel value after Laplace transformation is greater than the preset pixel threshold, determine the zoom direction of the adjusted focal length; control the focal length of the binocular camera to zoom in the zoom direction so that the edge values of multiple spans of the image in the area of interest meet the preset standard.
[0111] In an embodiment of the present invention, after performing Laplace transform on the region of interest, the present invention adjusts the focal length of the binocular camera. Specifically, two focal length increase and decrease processes may be performed. After the first focal length increase, the focus may be close to the optimal position, but may not be accurate. The second focal length increase can further confirm the focus position and ensure the accuracy of the focus. If only the focal length is increased, the image may be overfocused, making the target blurred. By further reducing the focal length, the focus can be adjusted back to the optimal position to overcome this problem.
[0112] Then, after the focus adjustment process, it is determined whether the pixel value after Laplace transformation is greater than the preset pixel threshold, that is, whether the pixel value after Laplace transformation increases. If the pixel value after Laplace transformation increases, it means that the focus increase is effective and the corresponding zoom direction is determined. Among them, checking the pixel value after Laplace transformation when adjusting the focus is an indicator related to the clarity and contrast of the image. During the focusing process, by monitoring the Laplace value, the clarity of the image can be evaluated in real time, so as to dynamically adjust the focus to achieve the best clarity of the image.
[0113] Finally, the focal length of the binocular camera is controlled to zoom in the zoom direction so that the edge values of multiple spans of the image in the region of interest meet the preset standard. The preset standard includes a preset edge threshold. If the edge values of multiple spans of the image in the region of interest are all greater than the preset edge threshold, it means that the image in the region of interest is obvious and clear.
[0114] Figure 3 This is a schematic diagram of the structure of the object tracking device based on the binocular camera provided by the present invention. Figure 3 As shown, an object tracking device based on a binocular camera includes: an auxiliary image acquisition module, used to acquire multiple sets of captured image sets captured by the binocular camera; wherein each set of captured image sets includes a left view and a right view; a target tracking module, used to perform feature extraction on the left view to obtain a first image feature; perform feature extraction on the right view to obtain a second image feature; construct the first image feature and the second image feature into a voxel set; obtain the matching cost of each voxel in the voxel set; determine a region of interest based on the matching cost and matching conditions of each voxel in the voxel set; wherein the region of interest contains a target object; based on the image position of the target object, adjust the moving direction of the moving module so that the target object is located at the center of the picture when the binocular camera is shooting; wherein the binocular camera is arranged above the moving module, and the moving module drives the binocular camera to move.
[0115] The present invention provides an object tracking device based on a binocular camera, comprising: obtaining multiple sets of captured images captured by a binocular camera through an auxiliary image acquisition module, wherein each set of captured images includes a left view and a right view, a target tracking module extracts features of the left view to obtain a first image feature, extracts features of the right view to obtain a second image feature, constructs the first image feature and the second image feature into a voxel set, obtains the matching cost of each voxel in the voxel set, determines an area of interest according to the matching cost and matching conditions of each voxel in the voxel set, wherein the area of interest includes a target object, and adjusts the moving direction of a moving module based on the image position of the target object so that the target object is located at the center of the picture when the binocular camera is shooting, wherein the binocular camera is arranged above the moving module, and the moving module drives the binocular camera to move. A voxel set is generated through overlapping views of the binocular camera, and Fourier position coding is aligned and fused with pixel projection, and finally the tracking of the target is completed by calculating the matching cost and cosine similarity matching. Compared with the existing target tracking method, this method has the advantage of anti-occlusion. In addition, the present invention also evaluates the image quality through cross-order Fourier transform to guide the automatic focusing of the binocular camera, thereby improving the shooting quality of the binocular camera.
[0116] Figure 4 FIG. 1 is a schematic diagram of a structure of an object tracking device based on a binocular camera provided by another embodiment of the present invention. Figure 4 As shown, the target tracking module may include: a feature extraction unit, used to normalize each voxel in the voxel set to obtain each normalized voxel in the voxel set; perform Fourier position coding on each normalized voxel in the voxel set to obtain each encoded voxel in the voxel set; obtain each fused position code in the voxel set according to each encoded voxel in the voxel set; and a target matching unit, used to calculate the matching cost of each voxel in the voxel set according to each fused position code in the voxel set.
[0117] In one embodiment, the feature extraction unit can be specifically configured to: determine the body mass centroid in the voxel set; determine the projection value of the body mass centroid in the left view; determine the projection value of the body mass centroid in the right view; based on a multi-layer perceptron, generate each fusion position code in the voxel set according to the projection value of the body mass centroid in the left view, the projection value of the body mass centroid in the right view, and each encoded voxel in the voxel set.
[0118] In one embodiment, the target matching unit can be specifically configured as follows: based on the sampling condition, sampling the first image feature to obtain the sampled first image feature; based on the sampling condition, sampling the second image feature to obtain the sampled second image feature; performing feature splicing processing on each fusion position encoding, the sampled first image feature, and the sampled second image feature in the voxel set to obtain the spliced feature; based on the multi-layer perceptron, calculating the matching cost of each voxel in the voxel set according to the spliced feature.
[0119] In one embodiment, the target matching unit may be specifically configured to: sample the first image feature around each voxel in the voxel set and the body centroid of the voxel set with a preset step length to obtain the sampled first image feature.
[0120] In one embodiment, the target tracking module can be specifically configured as follows: downsampling the matching cost of each voxel in the voxel set to obtain a matching voxel; calculating the similarity between the matching voxels in the voxel set and the matching voxels in other voxel sets within a unit time step; and determining a region of interest based on multiple similarities.
[0121] In one embodiment, the target tracking module may be specifically configured to: select a plurality of matching voxels with similarities greater than a preset similarity threshold as target voxels; and combine the target voxels into a region of interest.
[0122] In one embodiment, the autofocus module can be specifically configured as follows: calculating edge values of multiple spans on the image within the region of interest; calculating the average value of the edge values of the multiple spans; if the average value of the edge values of the multiple spans is less than a preset average value threshold, performing Laplace transform on the pixel values of the image within the region of interest and focusing the focal length of the binocular camera based on the focusing scheme.
[0123] In one embodiment, the autofocus module can be specifically configured as follows: adjusting the focal length of the binocular camera and determining whether the pixel value after Laplace transformation is greater than a preset pixel threshold; if the pixel value after Laplace transformation is greater than the preset pixel threshold, determining the zoom direction of the adjusted focal length; controlling the focal length of the binocular camera to zoom in the zoom direction so that the edge values of multiple spans of the image within the region of interest meet the preset standard.
[0124] Figure 5 A block diagram of an electronic device according to an embodiment of the present application is illustrated.
[0125] like Figure 5 As shown, the electronic device 10 includes one or more processors 11 and a memory 12 .
[0126] The processor 11 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 10 to perform desired functions.
[0127] The memory 12 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 11 may run the program instructions to implement the object tracking method based on the binocular camera of each embodiment of the present application described above and / or other desired functions. Various contents such as input signals, signal components, noise components, etc. may also be stored in the computer-readable storage medium.
[0128] In one example, the electronic device 10 may further include: an input device 13 and an output device 14 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).
[0129] When the electronic device 10 is a stand-alone device, the input device 13 may be a communication network connector for receiving collected input signals from the first device and the second device.
[0130] In addition, the input device 13 may also include, for example, a keyboard, a mouse, and the like.
[0131] The output device 14 can output various information to the outside, including the determined distance information, direction information, etc. The output device 14 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.
[0132] Of course, to simplify, Figure 5 Only some of the components related to the present application in the electronic device 10 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device 10 may also include any other appropriate components.
[0133] The computer program product may be written in any combination of one or more programming languages to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.
[0134] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.
[0135] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.
Claims
1. An object tracking method based on a binocular camera, characterized in that: include: Acquire multiple sets of captured image sets captured by a binocular camera; wherein each set of captured image sets includes a left view and a right view; Performing feature extraction on the left view to obtain a first image feature; Performing feature extraction on the right view to obtain a second image feature; constructing the first image feature and the second image feature into a voxel set; Normalizing each voxel in the voxel set to obtain each normalized voxel in the voxel set; Performing Fourier position coding processing on each normalized voxel in the voxel set to obtain each encoded voxel in the voxel set; determining a centroid of a body mass in the set of voxels; Determine a projection value of the centroid of the body in the left view; Determine a projection value of the centroid of the body in the right view; Based on a multi-layer perceptron, a code for each fusion position in the voxel set is generated according to the projection value of the body mass centroid on the left view, the projection value of the body mass centroid on the right view, and each encoded voxel in the voxel set; wherein the calculation formula for the total projection value of the body mass centroid on the left view and the right view is: , where p represents the physical fitness center, represents the projection value of the centroid of the body on the left view, represents the projection value of the centroid of the body on the right view, represents the projection matrix of the centroid of the body in the left view and represents the projection matrix of the centroid of the body on the right view, represents the first image feature, The second image feature, each fusion position in the voxel set encodes The calculation formula is: ; Where q represents each encoded voxel in the voxel set; Based on the sampling condition, sampling processing is performed on the first image feature to obtain a sampled first image feature; Based on the sampling condition, sampling processing is performed on the second image feature to obtain a sampled second image feature; Perform feature splicing processing on each fusion position code in the voxel set, the sampled first image feature, and the sampled second image feature to obtain a spliced feature; Based on a multi-layer perceptron, according to the concatenated features, a matching cost of each voxel in the voxel set is calculated; Determine a region of interest according to a matching cost and a matching condition of each voxel in the voxel set; wherein the region of interest includes a target object; Based on the image position of the target object, the moving direction of the moving module is adjusted so that the target object is located at the center of the picture when the binocular camera is shooting; wherein the binocular camera is arranged above the moving module, and the moving module drives the binocular camera to move.
2. The object tracking method based on a binocular camera according to claim 1, characterized in that: The sampling process of the first image feature based on the sampling condition to obtain the sampled first image feature includes: The first image feature is sampled by surrounding each voxel in the voxel set with the body centroid of the voxel set at a preset step length to obtain the sampled first image feature.
3. The object tracking method based on a binocular camera according to claim 1, characterized in that: The determining of the region of interest according to the matching cost and matching condition of each voxel in the voxel set includes: Downsampling the matching cost of each voxel in the voxel set to obtain a matching voxel; Calculating the similarity between the matching voxels in the voxel set and the matching voxels in other voxel sets within a unit time step; Based on multiple similarities, the region of interest is determined.
4. The object tracking method based on a binocular camera according to claim 3, characterized in that: Determining the region of interest according to the multiple similarities includes: Selecting the plurality of matching voxels whose similarities are greater than a preset similarity threshold as target voxels; The target voxels are grouped into regions of interest.
5. The object tracking method based on a binocular camera according to claim 1, characterized in that: Before adjusting the moving direction of the moving module based on the image position of the target object, the method further includes: Calculate edge values of multiple spans on the image within the region of interest; Calculating an average of the edge values of the plurality of spans; If the average value of the edge values of the multiple spans is less than the preset average value threshold, Laplace transform is performed on the pixel values of the image in the region of interest, and the focus of the binocular camera is adjusted based on the focusing scheme.
6. The object tracking method based on a binocular camera according to claim 5, characterized in that: The focusing process of adjusting the focal length of the binocular camera based on the focusing scheme includes: Adjusting the focal length of the binocular camera and determining whether the pixel value after Laplace transformation is greater than a preset pixel threshold; If the pixel value after the Laplace transformation is greater than the preset pixel threshold, determining the zoom direction of the adjusted focal length; The focal length of the binocular camera is controlled to zoom in the zoom direction so that edge values of multiple spans of the image in the region of interest meet a preset standard.
7. An object tracking device based on a binocular camera, characterized in that: include: The auxiliary image acquisition module is used to acquire multiple sets of captured images captured by the binocular camera; wherein each set of captured images includes a left view and a right view; The target tracking module is used to extract features from the left view to obtain a first image feature; extract features from the right view to obtain a second image feature; construct the first image feature and the second image feature into a voxel set; normalize each voxel in the voxel set to obtain each normalized voxel in the voxel set; perform Fourier position encoding on each normalized voxel in the voxel set to obtain each encoded voxel in the voxel set; determine the body mass center in the voxel set; determine the projection value of the body mass center in the left view; determine the projection value of the body mass center in the right view; based on a multi-layer perceptron, generate each fusion position code in the voxel set according to the projection value of the body mass center in the left view, the projection value of the body mass center in the right view and each encoded voxel in the voxel set; wherein the calculation formula for the total projection value of the body mass center on the left view and the right view is: , where p represents the physical fitness center, represents the projection value of the centroid of the body on the left view, represents the projection value of the centroid of the body on the right view, represents the projection matrix of the centroid of the body in the left view and represents the projection matrix of the centroid of the body on the right view, represents the first image feature, The second image feature, each fusion position in the voxel set encodes The calculation formula is: ; wherein q represents each encoded voxel in the voxel set; based on the sampling condition, the first image feature is sampled to obtain the sampled first image feature; based on the sampling condition, the second image feature is sampled to obtain the sampled second image feature; each fusion position encoding in the voxel set, the sampled first image feature, and the sampled second image feature are subjected to feature splicing processing to obtain the spliced feature; based on the multi-layer perceptron, the matching cost of each voxel in the voxel set is calculated according to the spliced feature; the region of interest is determined according to the matching cost of each voxel in the voxel set and the matching condition; wherein the region of interest contains the target object; based on the image position of the target object, the moving direction of the moving module is adjusted so that the target object is located at the center of the picture when the binocular camera is shooting; wherein the binocular camera is arranged above the moving module, and the moving module drives the binocular camera to move.
