A target tracking method based on binocular camera and related device

Abstract

The present application discloses a target tracking method based on a binocular camera and a related device, the method comprising: using a binocular camera to obtain a video frame of a target object; performing cross-section processing on a preset target three-dimensional model according to the video frame to obtain a target object mask, the preset target three-dimensional model is a three-dimensional modeling corresponding to the target object; extracting the running corner point parameters of the target object through the target object mask; using a preset sparse optical flow method to perform real-time target tracking on the target object according to the running corner point parameters. The present application solves the technical problem that the existing target tracking technology is easily affected by environmental changes, resulting in poor tracking stability and accuracy.

Description

Technical Field

[0001] The present application relates to the field of target tracking technology, and in particular to a target tracking method based on a binocular camera and related devices. Background Art

[0002] Target tracking is widely used in industrial production, robotics and other fields. With the development of drone technology, the demand for tracking ground targets, especially ground moving targets, is increasing. Tracking needs are mainly divided into two stages: the discovery of ground targets and long-term locking. In the long-term target locking stage, since the drone platform itself is in an incompletely controllable environment affected by external conditions, there have always been technical difficulties in the long-term tracking of ground targets, especially ground moving targets by drones. Existing target tracking technology is easily affected by the environment, resulting in poor tracking stability and lack of accuracy in tracking results. Summary of the invention

[0003] The present application provides a target tracking method and related devices based on a binocular camera, which are used to solve the technical problem that the existing target tracking technology is easily affected by environmental changes, resulting in poor tracking stability and accuracy.

[0004] In view of this, the first aspect of the present application provides a target tracking method based on a binocular camera, comprising:

[0005] A binocular camera is used to obtain video frames of the target object;

[0006] Performing cross-section processing on a preset target three-dimensional model according to the video frame to obtain a target object mask, wherein the preset target three-dimensional model is a three-dimensional modeling corresponding to the target object;

[0007] Extracting the running corner point parameters of the target object through the target object mask;

[0008] A preset sparse optical flow method is used to perform real-time target tracking on the target object according to the running corner point parameters.

[0009] Optionally, performing cross-section processing on a preset target three-dimensional model according to the video frame to obtain a target object mask, wherein the preset target three-dimensional model is three-dimensional modeling corresponding to the target object, includes:

[0010] Adjusting a preset target three-dimensional model to be consistent with the position and size of the target object in the video frame based on a preset adjustment operation, wherein the preset target three-dimensional model is a three-dimensional model corresponding to the target object;

[0011] Cross-section processing is performed on the preset target three-dimensional model to obtain a target object mask, wherein the target object mask includes size information, shape information and position information.

[0012] Optionally, extracting the running corner point parameters of the target object through the target object mask includes:

[0013] Detecting the running corner points of the target object in the video frame by using the target object mask to obtain strong feature points;

[0014] The running corner point parameters of the target object are calculated according to the strong feature points.

[0015] Optionally, the method of using a preset sparse optical flow method to perform real-time target tracking on the target object according to the running corner point parameters further includes:

[0016] Using a depth camera to obtain the mean coordinate point of the target object according to the tracking result;

[0017] The Z-axis data of the corresponding points are obtained based on the mean coordinate points, and integrated to obtain a multi-degree-of-freedom array.

[0018] The second aspect of the present application provides a target tracking device based on a binocular camera, comprising:

[0019] A target acquisition module, used to acquire a video frame of a target object using a binocular camera;

[0020] A cross-section processing module, used for performing cross-section processing on a preset target three-dimensional model according to the video frame to obtain a target object mask, wherein the preset target three-dimensional model is a three-dimensional model corresponding to the target object;

[0021] A parameter extraction module, used to extract the running corner point parameters of the target object through the target object mask;

[0022] The real-time tracking module is used to perform real-time target tracking on the target object according to the running corner point parameters by using a preset sparse optical flow method.

[0023] Optionally, the cross-section processing module is specifically used to:

[0024] Adjusting a preset target three-dimensional model to be consistent with the position and size of the target object in the video frame based on a preset adjustment operation, wherein the preset target three-dimensional model is a three-dimensional model corresponding to the target object;

[0025] Cross-section processing is performed on the preset target three-dimensional model to obtain a target object mask, wherein the target object mask includes size information, shape information and position information.

[0026] Optionally, the parameter extraction module is specifically used to:

[0027] Detecting the running corner points of the target object in the video frame by using the target object mask to obtain strong feature points;

[0028] The running corner point parameters of the target object are calculated according to the strong feature points.

[0029] Optionally, also include:

[0030] A multi-free array acquisition module, used to acquire the mean coordinate point of the target object according to the tracking result using a depth camera;

[0031] The Z-axis data of the corresponding points are obtained based on the mean coordinate points, and integrated to obtain a multi-degree-of-freedom array.

[0032] The third aspect of the present application provides a target tracking device based on a binocular camera, the device comprising a processor and a memory;

[0033] The memory is used to store program code and transmit the program code to the processor;

[0034] The processor is used to execute the target tracking method based on a binocular camera described in the first aspect according to the instructions in the program code.

[0035] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store program code, and the program code is used to execute the target tracking method based on a binocular camera described in the first aspect.

[0036] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0037] In the present application, a target tracking method based on a binocular camera is provided, including: using a binocular camera to obtain a video frame of a target object; performing cross-section processing on a preset target three-dimensional model according to the video frame to obtain a target object mask, wherein the preset target three-dimensional model is a three-dimensional modeling corresponding to the target object; extracting the running corner point parameters of the target object through the target object mask; and using a preset sparse optical flow method to perform real-time target tracking on the target object according to the running corner point parameters.

[0038] The target tracking method based on binocular camera provided by the present application uses binocular camera to obtain the video frame of the target object, calculates the accurate position of the target object through the internal baseline distance of the binocular camera, and is not affected by the environment in which the target object is located, so as to accurately locate the target object; and the calculation of relevant parameters based on the three-dimensional model corresponding to the target object is also more conducive to accurately describing the shape, size and position of the target object, which is helpful for precise positioning and tracking. Therefore, the present application can solve the technical problem that the existing target tracking technology is easily affected by environmental changes, resulting in poor tracking stability and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A schematic diagram of a process of a target tracking method based on a binocular camera provided in an embodiment of the present application;

[0040] Figure 2 A schematic diagram of the structure of a target tracking device based on a binocular camera provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0042] For easier understanding, see Figure 1 , the present application provides an embodiment of a target tracking method based on a binocular camera, comprising:

[0043] Step 101: Use a binocular camera to acquire a video frame of a target object.

[0044] A stereo camera consists of two monocular cameras, but the distance between the two cameras, i.e. the baseline, is known. The spatial position of each pixel is estimated through this baseline. The larger the baseline distance, the farther the distance that can be measured.

[0045] By calculating the parallax of the two images, the distance to the scene ahead, i.e. the target object, can be measured directly without judging what type of obstacle is ahead. Therefore, for any type of obstacle, necessary warnings or braking can be performed according to the change in distance information. The principle of binocular cameras is similar to that of human eyes; the human eye can perceive the distance of an object because the two eyes present different images of the same object, also known as "parallax". The farther the object is, the smaller the parallax; conversely, the larger the parallax. The size of the parallax corresponds to the distance between the object and the eye.

[0046] The configuration and calibration of binocular cameras are relatively complex, and their depth range and accuracy are limited by the baseline and resolution of the binoculars. In addition, visual computing consumes a lot of computing resources and requires GPU and FPGA devices to accelerate in order to output the distance information of the entire image in real time. Therefore, this application also needs to use other related parameters and the video frames obtained by the binocular camera to jointly realize the positioning of the target, reduce the amount of calculation while ensuring the accuracy of target tracking.

[0047] Step 102: Perform cross-section processing on the preset target three-dimensional model according to the video frame to obtain a target object mask, where the preset target three-dimensional model is the three-dimensional modeling corresponding to the target object.

[0048] Further, step 102 includes:

[0049] Based on the preset adjustment operation, the preset target three-dimensional model is adjusted to be consistent with the position and size of the target object in the video frame, and the preset target three-dimensional model is the three-dimensional modeling corresponding to the target object;

[0050] Cross-section processing is performed on a preset target three-dimensional model to obtain a target object mask, which includes size information, shape information, and position information.

[0051] The preset target 3D model is constructed in advance for the target object. The specific 3D model construction can be completed by referring to the existing technology and will not be described in detail here. Before cross-section processing, the preset target 3D model needs to be adjusted to be consistent with the shape of the target object in the video frame, including the shape, size, tilt angle, and specific position in the background of the video frame, etc. Then, the surface presented by the target object in the video frame is cut out in the 3D model, and finally these cut surfaces are combined to obtain the target object mask. The target object mask is similar to a mask, but it is a shape with surface features extracted from the 3D model.

[0052] Step 103: extract the running corner point parameters of the target object through the target object mask.

[0053] Further, step 103 includes:

[0054] The running corner points of the target object in the video frame are detected by using the target object mask to obtain strong feature points;

[0055] Calculate the running corner point parameters of the target object based on the strong feature points.

[0056] The target object mask is covered on the target object in the video frame, the running corner points of the target object are detected and calculated, the key corner points of the target object are obtained, and the feature points of the key corner points, that is, strong feature points, are extracted; then the running corner point parameters of the target object are calculated based on the strong feature points, and the target object tracking is started based on this parameter or as an initial parameter.

[0057] Step 104: Use a preset sparse optical flow method to perform real-time target tracking on the target object according to the running corner point parameters.

[0058] The sparse optical flow method is a method specifically used for image registration for sparse points on an image. That is, given a number of points (usually corner points) on a reference image, find their corresponding points in the current image. After confirming the corner points, all corner point data, that is, the running corner point parameters will be used as the initial point of the optical flow algorithm. After the optical flow algorithm is run, the latest positions of all tracking points will be updated in real time in the video screen. The algorithm has a small amount of calculation, which is conducive to reducing calculation pressure.

[0059] Furthermore, step 104 further includes:

[0060] Use a depth camera to obtain the mean coordinate points of the target object based on the tracking results;

[0061] The Z-axis data of the corresponding points are obtained based on the mean coordinate points, and integrated to obtain a multi-degree-of-freedom array.

[0062] It can be understood that the method provided in this embodiment is based on tracking of two-dimensional images. If you want to further improve the tracking accuracy, you can also use the camera of the depth camera to obtain the mean coordinate point based on the two-dimensional tracking results, and then obtain the Z-axis data corresponding to the target object based on the mean coordinate point. The three-dimensional coordinates can be integrated with the two-dimensional coordinates X-axis and Y-axis to obtain the three-axis posture information of α, β, and γ. The three-dimensional model of the target is attached to the target object to obtain the three-axis posture information of α, β, and γ. The integration with the three-axis posture information can obtain a 6-degree-of-freedom array, and the position information of the target object is expressed by different parameters, which is more accurate and reliable.

[0063] The target tracking method based on binocular camera provided in the embodiment of the present application uses binocular camera to obtain the video frame of the target object, calculates the accurate position of the target object through the internal baseline distance of the binocular camera, is not affected by the environment in which the target object is located, and accurately locates the target object; and the calculation of relevant parameters based on the three-dimensional model corresponding to the target object is also more conducive to accurately describing the shape, size and position of the target object, which is helpful for precise positioning and tracking. Therefore, the embodiment of the present application can solve the technical problem that the existing target tracking technology is easily affected by environmental changes, resulting in poor tracking stability and accuracy.

[0064] For easier understanding, see Figure 2 The present application provides an embodiment of a target tracking device based on a binocular camera, comprising:

[0065] A target acquisition module 201 is used to acquire a video frame of a target object using a binocular camera;

[0066] A cross-section processing module 202 is used to perform cross-section processing on a preset target three-dimensional model according to a video frame to obtain a target object mask, where the preset target three-dimensional model is a three-dimensional model corresponding to the target object;

[0067] A parameter extraction module 203 is used to extract the running corner point parameters of the target object through the target object mask;

[0068] The real-time tracking module 204 is used to perform real-time target tracking on the target object according to the running corner point parameters using a preset sparse optical flow method.

[0069] Furthermore, the cross-section processing module 202 is specifically configured to:

[0070] Based on the preset adjustment operation, the preset target three-dimensional model is adjusted to be consistent with the position and size of the target object in the video frame, and the preset target three-dimensional model is the three-dimensional modeling corresponding to the target object;

[0071] Cross-section processing is performed on a preset target three-dimensional model to obtain a target object mask, which includes size information, shape information, and position information.

[0072] Furthermore, the parameter extraction module 203 is specifically used for:

[0073] The running corner points of the target object in the video frame are detected by using the target object mask to obtain strong feature points;

[0074] Calculate the running corner point parameters of the target object based on the strong feature points.

[0075] Furthermore, it also includes:

[0076] A multi-free array acquisition module 205 is used to acquire the mean coordinate point of the target object according to the tracking result using a depth camera;

[0077] The Z-axis data of the corresponding points are obtained based on the mean coordinate points, and integrated to obtain a multi-degree-of-freedom array.

[0078] The present application also provides a target tracking device based on a binocular camera, the device comprising a processor and a memory;

[0079] The memory is used to store the program code and transmit the program code to the processor;

[0080] The processor is used to execute the target tracking method based on the binocular camera in the above method embodiment according to the instructions in the program code.

[0081] The present application also provides a computer-readable storage medium, which is used to store program codes, and the program codes are used to execute the target tracking method based on a binocular camera in the above method embodiment.

[0082] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0083] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0084] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0085] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for executing all or part of the steps of the method described in each embodiment of the present application through a computer device (which can be a personal computer, a server, or a network device, etc.). The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (full name in English: Read-Only Memory, English abbreviation: ROM), random access memory (full name in English: Random Access Memory, English abbreviation: RAM), disk or optical disk and other media that can store program codes.

[0086] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A target tracking method based on a binocular camera, characterized in that: include: A binocular camera is used to obtain video frames of the target object; According to the video frame, a preset target three-dimensional model is subjected to cross-section processing to obtain a target object mask, wherein the preset target three-dimensional model is a three-dimensional modeling corresponding to the target object, and the specific process is as follows: Adjusting a preset target three-dimensional model to be consistent with the position and size of the target object in the video frame based on a preset adjustment operation, wherein the preset target three-dimensional model is a three-dimensional model corresponding to the target object; Performing cross-section processing on the preset target three-dimensional model to obtain a target object mask, wherein the target object mask includes size information, shape information, and position information; The running corner point parameters of the target object are extracted through the target object mask, and the specific process is as follows: Detecting the running corner points of the target object in the video frame by using the target object mask to obtain strong feature points; Calculating the running corner point parameters of the target object according to the strong feature points; A preset sparse optical flow method is used to perform real-time target tracking on the target object according to the running corner point parameters.

2. The target tracking method based on a binocular camera according to claim 1, characterized in that: The method further comprises: performing real-time target tracking on the target object according to the running corner point parameters by using a preset sparse optical flow method; and then: Using a depth camera to obtain the mean coordinate point of the target object according to the tracking result; The Z-axis data of the corresponding points are obtained based on the mean coordinate points, and integrated to obtain a multi-degree-of-freedom array.

3. A target tracking device based on a binocular camera, characterized in that: include: A target acquisition module, used to acquire a video frame of a target object using a binocular camera; A cross-section processing module is used to perform cross-section processing on a preset target three-dimensional model according to the video frame to obtain a target object mask, wherein the preset target three-dimensional model is a three-dimensional model corresponding to the target object, and the cross-section processing module is specifically used to: Adjusting a preset target three-dimensional model to be consistent with the position and size of the target object in the video frame based on a preset adjustment operation, wherein the preset target three-dimensional model is a three-dimensional model corresponding to the target object; Performing cross-section processing on the preset target three-dimensional model to obtain a target object mask, wherein the target object mask includes size information, shape information, and position information; A parameter extraction module is used to extract the running corner point parameters of the target object through the target object mask. The parameter extraction module is specifically used to: Detecting the running corner points of the target object in the video frame by using the target object mask to obtain strong feature points; Calculating the running corner point parameters of the target object according to the strong feature points; The real-time tracking module is used to perform real-time target tracking on the target object according to the running corner point parameters by using a preset sparse optical flow method.

4. The target tracking device based on a binocular camera according to claim 3, characterized in that: Also includes: A multi-free array acquisition module, used to acquire the mean coordinate point of the target object according to the tracking result using a depth camera; The Z-axis data of the corresponding points are obtained based on the mean coordinate points, and integrated to obtain a multi-degree-of-freedom array.

5. A target tracking device based on a binocular camera, characterized in that: The device comprises a processor and a memory; The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the target tracking method based on a binocular camera according to any one of claims 1-2 according to the instructions in the program code.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program code, and the program code is used to execute the target tracking method based on a binocular camera according to any one of claims 1-2.

Images