A target tracking method, device, equipment and medium

Abstract

The disclosed embodiments relate to a target tracking method, device, equipment and medium, wherein the method comprises: obtaining a target video frame; determining the position information of the target area in the target video frame by detecting the features of the target video frame; if the position information of the target area in the target video frame fails to be determined, determining the target shooting position information; and re-determining the position information of the target area in the target video frame according to the target shooting position information and the camera projection algorithm. By adopting the above technical solution, after the tracking of the target area of ​​the video frame fails, the position of the target area in the video frame can be re-determined according to the shooting position determined by the synchronous positioning and mapping algorithm to achieve tracking recovery, and target tracking can be achieved even when the camera moves faster, thereby improving the tracking robustness.

Description

Technical Field

[0001] The present disclosure relates to the field of video processing technology, and in particular to a target tracking method, device, equipment and medium. Background Art

[0002] With the continuous development of smart terminal technology, the demand for video content recognition and tracking is increasing.

[0003] Currently, edge features can be used to track targets in each video frame. However, when the camera moves quickly, the above method will fail to detect the target and cause tracking failure, and has poor robustness. Summary of the invention

[0004] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a target tracking method, device, equipment and medium.

[0005] The present disclosure provides a target tracking method, the method comprising:

[0006] Get the target video frame;

[0007] Determine the position information of the target area in the target video frame by detecting the features of the target video frame;

[0008] If the position information of the target area in the target video frame fails to be determined, determining the target shooting position information;

[0009] According to the target shooting position information and the camera projection algorithm, the position information of the target area in the target video frame is determined again.

[0010] The present disclosure also provides a target tracking device, the device comprising:

[0011] A video frame module, used to obtain a target video frame;

[0012] A first location module, configured to determine location information of a target area in the target video frame by detecting features of the target video frame;

[0013] A shooting position module, configured to determine target shooting position information if the position information of the target area in the target video frame fails to be determined;

[0014] The second position module is used to determine the position information of the target area in the target video frame again according to the target shooting position information and the camera projection algorithm.

[0015] An embodiment of the present disclosure also provides an electronic device, which includes: a processor; a memory for storing executable instructions of the processor; the processor is used to read the executable instructions from the memory and execute the instructions to implement the target tracking method provided by the embodiment of the present disclosure.

[0016] The embodiment of the present disclosure further provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is used to execute the target tracking method provided by the embodiment of the present disclosure.

[0017] The technical solution provided by the embodiment of the present disclosure has the following advantages over the prior art: the target tracking solution provided by the embodiment of the present disclosure obtains the target video frame; determines the position information of the target area in the target video frame by detecting the features of the target video frame; if the position information of the target area in the target video frame fails to be determined, the target shooting position information is determined; based on the target shooting position information and the camera projection algorithm, the position information of the target area in the target video frame is determined again. With the above technical solution, after the target area tracking of the video frame fails, the position of the target area in the video frame can be re-determined based on the shooting position determined by the synchronous positioning and mapping algorithm to achieve tracking recovery, and target tracking can be achieved even when the camera moves faster, thereby improving tracking robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0019] Figure 1 A flowchart of a target tracking method provided by an embodiment of the present disclosure;

[0020] Figure 2 A schematic diagram of a normal vector projection provided in an embodiment of the present disclosure;

[0021] Figure 3 A schematic diagram of another normal vector projection provided in an embodiment of the present disclosure;

[0022] Figure 4 A flowchart of another target tracking method provided by an embodiment of the present disclosure;

[0023] Figure 5 A flowchart of another target tracking method provided by an embodiment of the present disclosure;

[0024] Figure 6 A schematic diagram of the structure of a target tracking device provided by an embodiment of the present disclosure;

[0025] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0027] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0028] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0029] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0030] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0031] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0032] Figure 1 The present invention provides a flow chart of a target tracking method provided by an embodiment of the present invention. The method can be performed by a target tracking device, wherein the device can be implemented by software and / or hardware and can generally be integrated in an electronic device. Figure 1 As shown, the method includes:

[0033] Step 101: Obtain a target video frame.

[0034] A video frame is also called an image frame, which can be the smallest unit of a video or a single image. The target video frame can be any image frame that needs to be detected and tracked, a video frame in a video captured by a device with a video acquisition function or a video frame obtained from the Internet, or an image captured in real time, without limitation.

[0035] Step 102: Determine the position information of the target area in the target video frame by detecting the features of the target video frame.

[0036] The target area refers to the area where the target-shaped object is located in the target video frame. The target shape is not limited, for example, the target shape may include an ellipse, a circle, and a rectangle, etc. The disclosed embodiment takes the target area as an ellipse area in the target video frame as an example for explanation. The position information of the target area may be information that can characterize the position of the target area in the video frame, and may specifically include vertex coordinates, center point coordinates, and other information of the target area in the video frame.

[0037] In the disclosed embodiment, after acquiring the target video frame, a preset detection algorithm may be used to determine the position information of the target area in the target video frame. The preset detection algorithm may be a detection algorithm based on deep learning or a contour detection algorithm, etc., which may be specifically determined according to actual conditions. For example, when the target area is an elliptical area, the preset detection algorithm may be any ellipse detection algorithm, and the process of determining the position information of the target area in the video frame may include: using an ellipse detection algorithm to perform contour detection on the first video frame, and then fitting the ellipse contour obtained by the contour detection to obtain the position information of the elliptical area in the video frame.

[0038] Optionally, when the target video frame is a video frame in the target video, determining the position information of the target area in the target video frame may also include: extracting a first image in the target video and determining first position information of the target area in the first image; performing optical flow tracking on the target video frame according to initial feature points determined by the first position information to obtain target feature points; wherein the target video frame is an adjacent video frame of the first image in the target video; and fitting the target feature points to obtain position information of the target area in the target video frame.

[0039] The target video frame may be any image frame in the target video, and the first image may be the previous image frame in the target video that is adjacent to the target video frame in the time sequence. In the embodiment of the present disclosure, the above-mentioned preset detection algorithm may be used to detect the target area of ​​the first image, and determine the first position information of the target area in the first image. Determining the initial feature points according to the first position information includes: sampling the edge contour of the target area in the first image according to the first position information to determine the initial feature points. Optionally, sampling the edge contour of the target area in the first image according to the first position information to determine the initial feature points includes: when the target area is an elliptical area, representing the target area in polar coordinates according to the first position information to obtain an elliptical contour; wherein the first position information includes the vertex coordinates and / or center point coordinates of the target area in the first image; sampling in the elliptical contour according to the preset polar angle interval to obtain the initial feature points. Afterwards, the initial feature points obtained by the above sampling are tracked using the optical flow tracking algorithm, and the feature points that are successfully tracked are retained as the target feature points, and the feature points that fail to be tracked are eliminated. Fitting the target feature points to obtain the position information of the target area in the target video frame.

[0040] In some embodiments, the target feature points are fitted to obtain the position information of the target area in the target video frame, including: if the coverage range of the target feature points on the edge contour of the target area is greater than or equal to the preset range, the target feature points are fitted to obtain the position information of the target area in the target video frame. The preset range refers to a pre-set range that satisfies the shape of the target area, which can be set according to actual conditions. For example, the preset range can be 3 / 4 of the entire range of the edge contour. Specifically, after determining the target feature points, it can be determined whether the coverage range of the target points on the edge contour of the target area is greater than or equal to the preset range. If so, the fitting algorithm is used to fit the target feature points to obtain the position information of the target area in the target video frame. If the coverage range of the target feature points on the edge contour of the target area is less than the preset range, the preset detection algorithm can be directly used to detect the target video frame to determine the position information of the target area in the target video frame.

[0041] In some embodiments, after determining the first position information of the target area in the first image, it also includes: determining the change parameter of the target video frame relative to the first image; performing optical flow tracking on the target video frame according to the initial feature points determined by the first position information to obtain the target feature points, including: if it is determined based on the change parameter that the target does not meet the multiplexing condition, then performing optical flow tracking on the target video frame according to the initial feature points determined by the first position information to obtain the target feature points.

[0042] Among them, the transformation parameter refers to a parameter that characterizes the change of the target video frame relative to the first image. Optionally, determining the change parameter of the target video frame relative to the first image may include: extracting the first feature point in the first image; performing optical flow tracking on the target video frame according to the first feature point, determining the second feature point, and determining the moving distance between the second feature point and the first feature point as the change parameter. The first feature point may be a corner point detected by the first image using the FAST corner point detection algorithm. The reuse condition refers to the specific judgment condition for determining whether the first image can be reused based on the position of the target video frame relative to the target area. Among them, the change threshold refers to a pre-set threshold, which can be set according to actual conditions. For example, when the change parameter is characterized by the movement information of the feature point in the target video frame relative to the corresponding feature point in the first video, the transformation threshold may be set to 0.8 as the distance threshold.

[0043] Specifically, after determining the change parameter of the target video frame relative to the first image, the change parameter can be compared with the change threshold. If it is determined that the change parameter is greater than the change threshold, it can be determined that the target video frame does not meet the multiplexing condition and needs to be re-tracked. The optical flow tracking of the target video frame is performed based on the initial feature points determined according to the first position information to obtain the target feature points. Otherwise, it is determined that the target video frame meets the multiplexing condition, and the first position information is determined as the position information of the target area in the target video frame.

[0044] In the above scheme, based on the target area detection of an image frame of the video, the position of the target area in the target video frame can be more accurately determined by tracking and fitting the feature points, thereby improving the computational efficiency of determining the position of the target area in the target video frame. In addition, by adding the judgment of the reuse condition to two adjacent video frames, when the change between the two adjacent video frames in the video is large, the above feature point tracking and fitting are used to determine the position of the target area; when the change or difference between the two adjacent video frames in the video is small, it means that the two video frames are highly similar. At this time, the next video frame can directly reuse the position information of the target area of ​​the previous video frame without re-detection, which saves workload and improves computational efficiency.

[0045] Step 103: If the position information of the target area in the target video frame fails to be determined, the target shooting position information is determined.

[0046] In the embodiment of the present disclosure, when the camera moves quickly, the position information of the target area may not be successfully determined in the target video frame, so the target shooting position information is determined. The target shooting position information refers to the coordinates of the shooting position in the world coordinate system.

[0047] In an embodiment of the present disclosure, determining the target shooting position information may include: determining the initial shooting position information in a first coordinate system through a synchronous positioning and mapping algorithm; determining the target transformation relationship between the first coordinate system and a second coordinate system corresponding to a camera projection algorithm; and determining the target shooting position information based on the initial shooting position information and the target transformation relationship.

[0048] Among them, the Simultaneous Localization And Mapping (SLAM) algorithm, SLAM is mainly used to solve the problem of positioning, navigation and map construction when a device equipped with a specific sensor is running in an unknown environment. If the sensor is a camera, the position of the camera, that is, the shooting position, can be determined. In the embodiment of the present disclosure, the SLAM algorithm is used to determine the initial shooting position information of the shooting position in the first coordinate system. The camera projection algorithm can be an algorithm for the pinhole projection model of the camera.

[0049] Optionally, determining the target transformation relationship between the first coordinate system and the second coordinate system corresponding to the camera projection algorithm may include: respectively determining first shooting position information and second shooting position information of the inspected video frame through a synchronous positioning and mapping algorithm and a camera projection algorithm, wherein the inspected video frame is a video frame that successfully tracks the target area; determining the target transformation relationship based on the first shooting position information and the second shooting position information of the inspected video frame.

[0050] The detected video frame refers to a video frame that has been successfully tracked, that is, a video frame that has successfully determined the position of the target area. Among them, the number of detected video frames is one or more. When the number of detected video frames is multiple, the target transformation relationship is the average value of the transformation relationship corresponding to each detected video frame, and the target transformation relationship includes a transformation scale and a transformation displacement. The target transformation relationship refers to the transformation relationship between the first coordinate system corresponding to the synchronous positioning and mapping algorithm and the second coordinate system corresponding to the camera projection algorithm. The target transformation relationship may include the transformation between the two coordinate systems in rotation, scale and displacement. Since the rotation of the two coordinate systems is the same, there is no need to perform a rotation transformation, that is, the target transformation relationship includes a transformation scale and a transformation displacement. When the number of detected video frames is multiple, a transformation relationship can be calculated for each detected video frame, so the target transformation relationship can be the average value of the transformation relationship corresponding to each detected video frame.

[0051] Specifically, for the inspected video frame, the first shooting position information can be determined by the synchronous positioning and mapping algorithm, and the second shooting position information can be determined by the camera projection algorithm. The first shooting position information and the second shooting position information are the position information of the shooting position in the first coordinate system and the second coordinate system. Then, the target transformation relationship between the first coordinate system and the second coordinate system can be determined by the least square method according to the first shooting position information and the second shooting position information. Exemplarily, the first shooting position information can be represented by slam_w_T_c, the second shooting position information can be represented by ellipse_w_T_c, and the target transformation relationship is represented by slam_w_T_c=ellipse_w_T_c*align_scale+align_translation, where align_scale represents the transformation scale of the two coordinate systems, and align_translation represents the transformation displacement of the two coordinate systems.

[0052] It can be understood that the above-mentioned first coordinate system can be a first world coordinate system corresponding to the synchronous positioning and mapping algorithm, and the above-mentioned second coordinate system can be a second world coordinate system corresponding to the camera projection algorithm or a spatial coordinate system with the target shape object as the origin. The origin and coordinate axis of the above-mentioned first world coordinate system and the second world coordinate system are different, and also need to be transformed.

[0053] Optionally, the target shooting position information is determined based on the initial shooting position information and the target transformation relationship, including: based on the target transformation relationship, the initial shooting position information is transformed from information of the first coordinate system to information of the second coordinate system to obtain the target shooting position information. After determining the above-mentioned target transformation relationship, the initial shooting position information can be transformed from information of the first coordinate system corresponding to the synchronous positioning and mapping algorithm to information of the second coordinate system corresponding to the camera projection algorithm through the target transformation relationship, and then the target shooting position information after the coordinate system transformation can be used to perform the subsequent step of determining the position information of the target area in the target video frame.

[0054] Step 104: Determine the position information of the target area in the target video frame again according to the target shooting position information and the camera projection algorithm.

[0055] In an embodiment of the present disclosure, based on the target shooting position information and the camera projection algorithm, the position information of the target area in the target video frame is determined again, which can include: performing position solving according to the target shooting position information and the position information of the target shape object corresponding to the target area, and determining the displacement information from the shooting position to the target shape object; wherein the target area is the area where the target shape object is located in the target video frame; and inputting the displacement information from the shooting position to the target shape object into the projection equation of the camera projection algorithm to determine the position information of the target area in the target video frame.

[0056] The position information of the target shape object can be a pre-set fixed value, which is a known quantity. For example, the origin of the world coordinate system can be set at the position of the target shape object, and the coordinates of the position information are (0, 0, 0). According to the position of the target shape object, the target shooting position and the transformation equation, the displacement from the target shooting position to the target shape object can be determined. The transformation equation can be expressed as W10=W20+W12, wherein W10 represents the shooting position information in the world coordinate system, W20 represents the position information of the target shape object in the world coordinate system, and W12 represents the displacement information from the target shooting position to the target area in the world coordinate system. The above three W10, W20 and W12 are all vectors in the world coordinate system, with direction and size. In this step, the target shooting position information W10 and the position information W12 of the target shape object after the above coordinate system transformation can be input into the transformation equation to obtain the displacement information W12 from the target shooting position to the target shape object.

[0057] The position information of the target area in the projection equation is related to the internal parameters of the shooting device, the rotation matrix from the coordinate system of the shooting position to the world coordinate system, and the position information of the origin of the world coordinate system in the coordinate system of the shooting position. The projection equation can be expressed as p=π[K(R12*W20+T)], where π represents the coefficient, K represents the internal parameters of the shooting device, which may specifically include some parameters of the shooting device such as the focal length of the shooting position and the distortion parameter, R12 represents the rotation matrix from the coordinate system of the shooting position to the world coordinate system, p represents the position information of the target area, and T represents the position information of the origin of the world coordinate system in the coordinate system of the shooting position. The above projection equation p=π[K(R12*W20+T)] is transformed to obtain W12=-1*ratio*R21*K-1*p, R21 represents the inverse matrix of the rotation matrix from the coordinate system of the shooting position to the world coordinate system, ratio=1 / π, T is a three-dimensional coordinate, the value on the right side of the equal sign in the projection equation is also a three-dimensional coordinate, p is a two-dimensional coordinate, and ratio can be the value of the last dimension in the three-dimensional coordinate. By converting the displacement information W12 from the target shooting position to the target shape object in the above world coordinate system into -1*ratio*R21*K-1*p, the position information p of the target area in the target video frame can be determined.

[0058] The target tracking solution provided by the embodiment of the present disclosure obtains a target video frame; determines the position information of the target area in the target video frame by detecting the features of the target video frame; if the position information of the target area in the target video frame fails to be determined, the target shooting position information is determined; and the position information of the target area in the target video frame is determined again based on the target shooting position information and the camera projection algorithm. With the above technical solution, after the tracking of the target area of ​​the video frame fails, the position of the target area in the video frame can be re-determined based on the shooting position determined by the synchronous positioning and mapping algorithm to achieve tracking recovery. The target tracking can also be achieved when the camera moves faster, thereby improving the tracking robustness.

[0059] In some embodiments, the target tracking method may further include: determining an initial normal vector of a plane where the target-shaped object corresponding to the target area is located, and determining a projection of the initial normal vector on a horizontal plane or a vertical plane as a target normal vector.

[0060] Among them, the initial normal vector refers to the normal vector of the plane where the target shape object is located in the space, and the target normal vector refers to the optimized normal vector. Specifically, after determining the position information of the target area in the target video frame and the adjacent video frame, the initial normal vector can be determined by decomposing the homography matrix between the two video frames. The above homography matrix is ​​calculated by sampling the points in the target area, and the decomposition method can adopt the singular value decomposition (SVD) method. After determining the initial normal vector, the projection of the initial normal vector on the horizontal plane or the vertical plane can be determined as the target normal vector. Exemplarily, the initial normal vector is (0.05, 0.03, 0.994), and the target normal vector can be (0, 0, 1).

[0061] For example, Figure 2 A schematic diagram of a normal vector projection provided in an embodiment of the present disclosure, Figure 3 A schematic diagram of another normal vector projection provided in an embodiment of the present disclosure, Figure 2 and Figure 3 The distribution represents the target normal vector after the initial normal vector is projected in the vertical and horizontal directions.

[0062] In the above scheme, it is assumed that the object is only located on the horizontal plane or the vertical plane. The optimized target normal vector can be obtained by projecting the initial normal vector, which avoids the error caused by assuming that the camera is an orthogonal projection model when calculating the normal vector, improves the accuracy of the normal of the plane where the object is located in space, and thus improves the accuracy of the special effects display based on the normal.

[0063] Figure 4A flowchart of another target tracking method provided in an embodiment of the present disclosure is provided. Based on the above embodiment, this embodiment further optimizes the above target tracking method.

[0064] like Figure 4 As shown, the method includes:

[0065] Step 201: Obtain a target video frame.

[0066] Step 202: Determine the position information of the target area in the target video frame by detecting the features of the target video frame.

[0067] Step 203: If the position information of the target area in the target video frame fails to be determined, the initial shooting position information in the first coordinate system is determined by a synchronous positioning and mapping algorithm.

[0068] Step 204: Determine a target transformation relationship between the first coordinate system and the second coordinate system corresponding to the camera projection algorithm.

[0069] Optionally, determining a target transformation relationship between a first coordinate system and a second coordinate system corresponding to a camera projection algorithm includes: respectively determining first shooting position information and second shooting position information of an inspected video frame by a synchronous positioning and mapping algorithm and a camera projection algorithm, wherein the inspected video frame is a video frame that successfully tracks a target area; determining a target transformation relationship based on the first shooting position information and the second shooting position information of the inspected video frame.

[0070] The number of the inspected video frames is one or more. When the number of the inspected video frames is multiple, the target transformation relationship is the average value of the transformation relationships corresponding to the inspected video frames, and the target transformation relationship includes a transformation scale and a transformation displacement.

[0071] Step 205: Determine the target shooting position information according to the initial shooting position information and the target transformation relationship.

[0072] Optionally, determining the target shooting position information according to the initial shooting position information and the target transformation relationship includes: transforming the initial shooting position information from information of a first coordinate system to information of a second coordinate system based on the target transformation relationship to obtain the target shooting position information.

[0073] Step 206: Determine the position information of the target area in the target video frame again according to the target shooting position information and the camera projection algorithm.

[0074] Optionally, the position information of the target area in the target video frame is determined again based on the target shooting position information and the camera projection algorithm, including: solving the position based on the target shooting position information and the position information of the target shape object corresponding to the target area, and determining the displacement information from the shooting position to the target shape object; wherein the target area is the area where the target shape object is located in the target video frame; and inputting the displacement information from the shooting position to the target shape object into the projection equation of the camera projection algorithm to determine the position information of the target area in the target video frame.

[0075] In some embodiments, the target tracking method may further include: determining an initial normal vector of a plane where the target-shaped object corresponding to the target area is located, and determining a projection of the initial normal vector on a horizontal plane or a vertical plane as a target normal vector.

[0076] Next, the target tracking method in the embodiment of the present disclosure is further described by a specific example. Figure 5A flow chart of another target tracking method provided for an embodiment of the present disclosure is provided. Taking the target area as an elliptical area and the target shaped object as a circular object as an example, the specific process may include: 1. Initialization: In the first several video frames of the target video, the SLAM algorithm is used to complete the calculation of the shooting position in the first coordinate system, represented by slam_w_T_c, and the camera projection algorithm is used to complete the calculation of the shooting position in the second coordinate system according to the known position of the elliptical area, represented by ellipse_w_T_c. The above-mentioned first coordinate system is the first world coordinate system, and the above-mentioned second coordinate system is the second world coordinate system or the circular object coordinate system. 2. Coordinate system alignment. The similarity transformation relationship of the known rotation between the first coordinate system and the second coordinate system is calculated using the least squares method. When the camera is an image acquisition module set in an intelligent terminal, the rotation can be obtained by the inertial measurement unit (IMU) of the intelligent terminal. The rotation of the two coordinate systems is the same, so the similarity transformation relationship includes the displacement and scale required for the similarity transformation, thereby completing the coordinate system alignment (alignment) between the two coordinate systems. 3. Tracking: When the current frame in the target video is blurred and the elliptical area cannot be detected due to the fast camera movement, the shooting position determined by the SLAM algorithm and the target transformation relationship between the two coordinate systems are used to determine the shooting position in the second coordinate system corresponding to the camera projection algorithm. The shooting position determined by the SLAM algorithm can be represented by slam_w_T_c, and the shooting position corresponding to the camera projection algorithm can be represented by ellipse_w_T_c. The target transformation relationship is expressed as slam_w_T_c = ellipse_w_T_c*align_scale+align_translation, where align_scale represents the transformation scale of the two coordinate systems, and align_translation represents the transformation displacement of the two coordinate systems. Then, the position of the elliptical area in the current frame is determined based on the shooting position in the second coordinate system, the projection equation corresponding to the camera projection algorithm, and the transformation equation.

[0077] Furthermore, the shooting position in the second coordinate system determined by the camera projection algorithm is inaccurate in some cases, resulting in a large error in the transformation relationship between the first coordinate system and the second coordinate system. In view of the above problem, two queues can be set during the entire tracking process to save the most recent set number of slam_w_T_c and ellipse_w_T_c, respectively, and the set number can be 10. When the current frame is blurred and the ellipse cannot be detected due to the fast camera movement, the transformation relationship between the two coordinate systems is determined based on the saved slam_w_T_c and ellipse_w_T_c, and the average value of the multiple transformation relationships is determined as the final target transformation relationship. Thus, the shooting position in the second coordinate system corresponding to the camera projection algorithm is determined according to the shooting position and the target transformation relationship determined by the SLAM algorithm, and then the position of the elliptical area in the current frame is determined according to the shooting position in the two coordinate systems.

[0078] The target tracking solution provided by the embodiment of the present disclosure obtains a target video frame; determines the position information of the target area in the target video frame by detecting the features of the target video frame; if the position information of the target area in the target video frame fails to be determined, the target shooting position information is determined; and the position information of the target area in the target video frame is determined again based on the target shooting position information and the camera projection algorithm. With the above technical solution, after the tracking of the target area of ​​the video frame fails, the position of the target area in the video frame can be re-determined based on the shooting position determined by the synchronous positioning and mapping algorithm to achieve tracking recovery. The target tracking can also be achieved when the camera moves faster, thereby improving the tracking robustness.

[0079] Figure 6 This is a schematic diagram of the structure of a target tracking device provided by an embodiment of the present disclosure. The device can be implemented by software and / or hardware and can generally be integrated into an electronic device. Figure 6 As shown, the device comprises:

[0080] The video frame module 301 is used to obtain a target video frame;

[0081] A first location module 302, configured to determine location information of a target area in the target video frame by detecting features of the target video frame;

[0082] A shooting position module 303, configured to determine target shooting position information if the position information of the target area in the target video frame fails to be determined;

[0083] The second position module 304 is used to determine the position information of the target area in the target video frame again according to the target shooting position information and the camera projection algorithm.

[0084] Optionally, the shooting position module 303 is used to:

[0085] Determine the initial shooting position information in the first coordinate system through a synchronous positioning and mapping algorithm;

[0086] Determine a target transformation relationship between the first coordinate system and a second coordinate system corresponding to the camera projection algorithm;

[0087] The target shooting position information is determined according to the initial shooting position information and the target transformation relationship.

[0088] Optionally, the shooting position module 303 is used to:

[0089] Determine the first shooting position information and the second shooting position information of the detected video frame respectively by the synchronous positioning and mapping algorithm and the camera projection algorithm, wherein the detected video frame is a video frame that successfully tracks the target area;

[0090] The target transformation relationship is determined according to the first shooting position information and the second shooting position information of the inspected video frame.

[0091] Optionally, the number of the inspected video frames is one or more. When the number of the inspected video frames is multiple, the target transformation relationship is an average value of the transformation relationships corresponding to the inspected video frames, and the target transformation relationship includes a transformation scale and a transformation displacement.

[0092] Optionally, the shooting position module 303 is used to:

[0093] The initial shooting position information is transformed from information of the first coordinate system to information of the second coordinate system based on the target transformation relationship to obtain the target shooting position information.

[0094] Optionally, the second location module 304 is used to:

[0095] Performing position solving according to the target shooting position information and the position information of the target shape object corresponding to the target area, and determining the displacement information from the shooting position to the target shape object; wherein the target area is the area where the target shape object is located in the target video frame;

[0096] The displacement information from the shooting position to the target shape object is input into the projection equation of the camera projection algorithm to determine the position information of the target area in the target video frame.

[0097] Optionally, the device further comprises a normal module, which is used to:

[0098] An initial normal vector of a plane where the target shape object corresponding to the target area is located is determined, and a projection of the initial normal vector on a horizontal plane or a vertical plane is determined as a target normal vector.

[0099] The target tracking device provided in the embodiments of the present disclosure can execute the target tracking method provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0100] The embodiments of the present disclosure also provide a computer program product, including a computer program / instruction, which, when executed by a processor, implements the target tracking method provided by any embodiment of the present disclosure.

[0101] Figure 7 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. Figure 7 , which shows a schematic diagram of the structure of an electronic device 400 suitable for implementing the embodiment of the present disclosure. The electronic device 400 in the embodiment of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0102] like Figure 7 As shown, the electronic device 400 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage device 408 into a random access memory (RAM) 403. In the RAM 403, various programs and data required for the operation of the electronic device 400 are also stored. The processing device 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0103] Typically, the following devices may be connected to the I / O interface 405: an input device 406 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 407 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 408 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 409. The communication device 409 may allow the electronic device 400 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 7The electronic device 400 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0104] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 409, or installed from the storage device 408, or installed from the ROM 402. When the computer program is executed by the processing device 401, the above-mentioned functions defined in the target tracking method of the embodiment of the present disclosure are executed.

[0105] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0106] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0107] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0108] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains a target video frame; determines the position information of a target area in the target video frame by detecting features of the target video frame; if the position information of the target area in the target video frame fails to be determined, determines the target shooting position information; and determines the position information of the target area in the target video frame again based on the target shooting position information and a camera projection algorithm.

[0109] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0110] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0111] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit does not, in some cases, constitute a limitation on the unit itself.

[0112] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0113] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0114] According to one or more embodiments of the present disclosure, the present disclosure provides a target tracking method, including:

[0115] Get the target video frame;

[0116] Determine the position information of the target area in the target video frame by detecting the features of the target video frame;

[0117] If the position information of the target area in the target video frame fails to be determined, determining the target shooting position information;

[0118] According to the target shooting position information and the camera projection algorithm, the position information of the target area in the target video frame is determined again.

[0119] According to one or more embodiments of the present disclosure, in the target tracking method provided by the present disclosure, the step of determining the target shooting position information includes:

[0120] Determine the initial shooting position information in the first coordinate system through a synchronous positioning and mapping algorithm;

[0121] Determine a target transformation relationship between the first coordinate system and a second coordinate system corresponding to the camera projection algorithm;

[0122] The target shooting position information is determined according to the initial shooting position information and the target transformation relationship.

[0123] According to one or more embodiments of the present disclosure, in the target tracking method provided by the present disclosure, determining the target transformation relationship between the first coordinate system and the second coordinate system corresponding to the camera projection algorithm includes:

[0124] Determine the first shooting position information and the second shooting position information of the detected video frame respectively by the synchronous positioning and mapping algorithm and the camera projection algorithm, wherein the detected video frame is a video frame that successfully tracks the target area;

[0125] The target transformation relationship is determined according to the first shooting position information and the second shooting position information of the inspected video frame.

[0126] According to one or more embodiments of the present disclosure, in the target tracking method provided by the present disclosure, the number of the inspected video frames is one or more. When the number of the inspected video frames is multiple, the target transformation relationship is the average value of the transformation relationships corresponding to each of the inspected video frames, and the target transformation relationship includes a transformation scale and a transformation displacement.

[0127] According to one or more embodiments of the present disclosure, in the target tracking method provided by the present disclosure, determining the target shooting position information according to the initial shooting position information and the target transformation relationship includes:

[0128] The initial shooting position information is transformed from information of the first coordinate system to information of the second coordinate system based on the target transformation relationship to obtain the target shooting position information.

[0129] According to one or more embodiments of the present disclosure, in the target tracking method provided by the present disclosure, the position information of the target area in the target video frame is determined again according to the target shooting position information and the camera projection algorithm, including:

[0130] Performing position solving according to the target shooting position information and the position information of the target shape object corresponding to the target area, and determining the displacement information from the shooting position to the target shape object; wherein the target area is the area where the target shape object is located in the target video frame;

[0131] The displacement information from the shooting position to the target shape object is input into the projection equation of the camera projection algorithm to determine the position information of the target area in the target video frame.

[0132] According to one or more embodiments of the present disclosure, the target tracking method provided by the present disclosure further includes:

[0133] An initial normal vector of a plane where the target shape object corresponding to the target area is located is determined, and a projection of the initial normal vector on a horizontal plane or a vertical plane is determined as a target normal vector.

[0134] According to one or more embodiments of the present disclosure, the present disclosure provides a target tracking device, including:

[0135] A video frame module, used to obtain a target video frame;

[0136] A first location module, configured to determine location information of a target area in the target video frame by detecting features of the target video frame;

[0137] A shooting position module, configured to determine target shooting position information if the position information of the target area in the target video frame fails to be determined;

[0138] The second position module is used to determine the position information of the target area in the target video frame again according to the target shooting position information and the camera projection algorithm.

[0139] According to one or more embodiments of the present disclosure, in the target tracking device provided by the present disclosure, the shooting position module is used to:

[0140] Determine the initial shooting position information in the first coordinate system through a synchronous positioning and mapping algorithm;

[0141] Determine a target transformation relationship between the first coordinate system and a second coordinate system corresponding to the camera projection algorithm;

[0142] The target shooting position information is determined according to the initial shooting position information and the target transformation relationship.

[0143] According to one or more embodiments of the present disclosure, in the target tracking device provided by the present disclosure, the shooting position module is used to:

[0144] Determine the first shooting position information and the second shooting position information of the detected video frame respectively by the synchronous positioning and mapping algorithm and the camera projection algorithm, wherein the detected video frame is a video frame that successfully tracks the target area;

[0145] The target transformation relationship is determined according to the first shooting position information and the second shooting position information of the inspected video frame.

[0146] According to one or more embodiments of the present disclosure, in the target tracking device provided by the present disclosure, the number of the inspected video frames is one or more. When the number of the inspected video frames is multiple, the target transformation relationship is the average value of the transformation relationships corresponding to each of the inspected video frames, and the target transformation relationship includes a transformation scale and a transformation displacement.

[0147] According to one or more embodiments of the present disclosure, in the target tracking device provided by the present disclosure, the shooting position module is used to:

[0148] The initial shooting position information is transformed from information of the first coordinate system to information of the second coordinate system based on the target transformation relationship to obtain the target shooting position information.

[0149] According to one or more embodiments of the present disclosure, in the target tracking device provided by the present disclosure, the second position module is used to:

[0150] Performing position solving according to the target shooting position information and the position information of the target shape object corresponding to the target area, and determining the displacement information from the shooting position to the target shape object; wherein the target area is the area where the target shape object is located in the target video frame;

[0151] The displacement information from the shooting position to the target shape object is input into the projection equation of the camera projection algorithm to determine the position information of the target area in the target video frame.

[0152] According to one or more embodiments of the present disclosure, in the target tracking device provided by the present disclosure, the device further includes a normal module, which is used to:

[0153] An initial normal vector of a plane where the target shape object corresponding to the target area is located is determined, and a projection of the initial normal vector on a horizontal plane or a vertical plane is determined as a target normal vector.

[0154] According to one or more embodiments of the present disclosure, the present disclosure provides an electronic device, including:

[0155] processor;

[0156] a memory for storing instructions executable by the processor;

[0157] The processor is used to read the executable instructions from the memory and execute the instructions to implement any target tracking method provided in the present disclosure.

[0158] According to one or more embodiments of the present disclosure, the present disclosure provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is used to execute any target tracking method provided by the present disclosure.

[0159] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0160] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0161] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A target tracking method, characterized in that: include: Get the target video frame; Determine the position information of the target area in the target video frame by detecting the features of the target video frame; If the position information of the target area in the target video frame fails to be determined, determining the target shooting position information, wherein the target shooting position information is the position information corresponding to the camera after movement; According to the target shooting position information and the camera projection algorithm, the position information of the target area in the target video frame is determined again, wherein the camera projection algorithm is an algorithm of the pinhole projection model of the camera; The determining of the target shooting position information includes: Determine the initial shooting position information in the first coordinate system through a synchronous positioning and mapping algorithm; Determine a target transformation relationship between the first coordinate system and a second coordinate system corresponding to the camera projection algorithm; Determining the target shooting position information according to the initial shooting position information and the target transformation relationship; The determining of the target transformation relationship between the first coordinate system and the second coordinate system corresponding to the camera projection algorithm includes: Determine the first shooting position information and the second shooting position information of the detected video frame respectively by the synchronous positioning and mapping algorithm and the camera projection algorithm, wherein the detected video frame is a video frame that successfully tracks the target area; The target transformation relationship is determined according to the first shooting position information and the second shooting position information of the inspected video frame.

2. The method according to claim 1, characterized in that The number of the inspected video frames is one or more. When the number of the inspected video frames is multiple, the target transformation relationship is an average value of the transformation relationships corresponding to the inspected video frames, and the target transformation relationship includes a transformation scale and a transformation displacement.

3. The method according to claim 1, characterized in that Determining the target shooting position information according to the initial shooting position information and the target transformation relationship includes: The initial shooting position information is transformed from information of the first coordinate system to information of the second coordinate system based on the target transformation relationship to obtain the target shooting position information.

4. The method according to claim 1, characterized in that: According to the target shooting position information and the camera projection algorithm, the position information of the target area in the target video frame is determined again, including: The position is solved according to the target shooting position information and the position information of the target shape object corresponding to the target area, and the displacement information from the shooting position to the target shape object is determined; wherein the target area is the area where the target shape object is located in the target video frame, and the position information of the target shape object is a preset fixed value; The displacement information from the shooting position to the target shape object is input into the projection equation of the camera projection algorithm to determine the position information of the target area in the target video frame.

5. The method according to claim 1, characterized in that Also includes: An initial normal vector of a plane where the target shape object corresponding to the target area is located is determined, and a projection of the initial normal vector on a horizontal plane or a vertical plane is determined as a target normal vector.

6. A target tracking device, characterized in that: include: A video frame module, used to obtain a target video frame; A first location module, configured to determine location information of a target area in the target video frame by detecting features of the target video frame; A shooting position module, used for determining target shooting position information if the position information of the target area in the target video frame fails to be determined, wherein the target shooting position information is the position information corresponding to the camera after movement; A second position module is used to determine the position information of the target area in the target video frame again according to the target shooting position information and a camera projection algorithm, wherein the camera projection algorithm is an algorithm of a pinhole projection model of a camera; The determining of the target shooting position information includes: Determine the initial shooting position information in the first coordinate system through a synchronous positioning and mapping algorithm; Determine a target transformation relationship between the first coordinate system and a second coordinate system corresponding to the camera projection algorithm; Determining the target shooting position information according to the initial shooting position information and the target transformation relationship; The determining of the target transformation relationship between the first coordinate system and the second coordinate system corresponding to the camera projection algorithm includes: Determine the first shooting position information and the second shooting position information of the detected video frame respectively by the synchronous positioning and mapping algorithm and the camera projection algorithm, wherein the detected video frame is a video frame that successfully tracks the target area; The target transformation relationship is determined according to the first shooting position information and the second shooting position information of the inspected video frame.

7. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the instructions to implement the target tracking method described in any one of claims 1-5 above.

8. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and the computer program is used to execute the target tracking method described in any one of claims 1 to 5.

Images