Planar image tracking methods, terminals, and storage media

Abstract

The present invention provides a planar image tracking method, a terminal and a storage medium. The logic control system generation method includes: S101: obtaining the global pose of the first three frames of images and the relative pose between two adjacent frames of images according to the corresponding relationship of the feature points of the target image in the first three frames of images, and obtaining the 3D coordinates of the feature points; S102: calculating the global pose of each frame of images after the first three frames of images and the relative pose between two adjacent frames of images according to the 3D coordinates, and optimizing the global pose and 3D coordinates; S103: respectively calculating the relative pose through the feature points and the global pose of the two adjacent frames of images, optimizing the relative pose according to the error of the relative pose, obtaining the optimal global pose through the relative pose, and using the global pose for real-time tracking. The present invention avoids the problem of using a large number of parameters for calculation and increasing the amount of calculation, has low requirements on hardware, realizes efficient and stable tracking of images, and reduces the cost of motion tracking.

Description

Technical Field

[0001] The present invention relates to the field of image tracking, and in particular to a planar image tracking method, a terminal and a storage medium. Background Art

[0002] With the development of computer vision technology, people's requirements for visual experience are getting higher and higher, and the rise of AR is undoubtedly a good way to meet people's needs. Among them, motion tracking based on plane images is one of the core technologies and applications of AR.

[0003] The steps of motion tracking of planar images mainly include detection and recognition, feature point matching, pose calculation and optimization. Deep learning is often used for motion tracking. In recent years, many good models have emerged in the field of deep learning, which have good effects in semantic understanding and semantic segmentation. However, due to the large number of parameters, deep neural network models require a large number of high-quality training and test sets, have large computational complexity, and high hardware requirements, which increases the cost of motion tracking and limits its use and development. Summary of the invention

[0004] In order to overcome the shortcomings of the prior art, the present invention proposes a planar image tracking method, a terminal and a storage medium. After shooting an image of a target image, the 3D coordinates of the feature points are obtained according to the feature points in the first three frames of the target image, and the global pose and relative pose of the subsequently shot images are calculated according to the 3D coordinates. The global pose is optimized by the error of the relative pose, and the motion tracking of the image is performed according to the optimized global pose, which avoids the need to use a large number of parameters for calculation and increases the amount of calculation. The hardware requirements are low and efficient and stable tracking of the image is achieved, which reduces the cost of motion tracking and expands the scope of use and development direction of motion tracking.

[0005] To solve the above problems, a technical solution adopted by the present invention is: a planar image tracking method, which includes: S101: selecting a target image, obtaining the global pose of the first three frames of images and the relative pose between two adjacent frames of images according to the correspondence between the feature points of the target image in the first three frames of images of the target image, and obtaining the 3D coordinates of the feature points through the global pose and adjacent pose; S102: calculating the global pose of each frame of images after the first three frames of images and the relative pose between two adjacent frames of images according to the 3D coordinates, and optimizing the global pose and 3D coordinates through a back projection error function; S103: calculating the relative pose through the feature points and global pose of two adjacent frames of images respectively, optimizing the relative pose according to the error of the relative pose, calculating the global pose of the image through the optimized relative pose, and using the global pose to perform real-time tracking and pose optimization on each frame of images.

[0006] Furthermore, the step of selecting the target image specifically includes: obtaining selection conditions of the target image, selecting the target image according to the selection conditions, and extracting and saving image coordinates and feature vectors of feature points in the target image.

[0007] Furthermore, the step of obtaining the global pose of the first three frames of images and the relative pose between two adjacent frames of images based on the correspondence between the feature points of the target image in the first three frames of images of the target image specifically includes: determining the coordinate system in which the shooting device is located when shooting the first frame of image as the world coordinate system, obtaining the correspondence between the feature points of the target image in the first three frames of images, and calculating the relative pose between two adjacent frames of images and the global pose of the second frame of image and the third frame of image relative to the world coordinate system based on the correspondence.

[0008] Furthermore, the step of obtaining the correspondence between the feature points of the target image in the first three frames of images specifically includes: extracting the feature points and feature vectors in the target image, matching the feature points and feature vectors with the feature points and feature vectors of the target image to calculate the homography matrix, and obtaining the correspondence between the feature points in the first three frames of images according to the homography matrix.

[0009] Furthermore, after the step of acquiring the 3D coordinates of the feature points through the global pose and the adjacent poses, the step further includes: optimizing the global pose and the 3D coordinates through a back-projection error function.

[0010] Furthermore, the step of calculating the global pose of each frame image after the first three frames of images and the relative pose between two adjacent frames of images according to the 3D coordinates specifically includes: generating 2D-3D point pairs according to the correspondence between the feature points of the current frame and the first frame and the 3D coordinates of the feature points, calculating the global pose of the current frame through the 2D-3D point pairs, and calculating the relative pose of the current frame and the previous frame through the correspondence between the feature points of the current frame and the previous frame.

[0011] Furthermore, after the step of optimizing the global pose and 3D coordinates through the back-projection error function, the step also includes: placing the current frame into a sliding window, and when the number of frames in the sliding window is greater than a preset value, removing the frame image with the smallest sequence number in the sliding window.

[0012] Furthermore, the step of optimizing the relative posture according to the error of the relative posture specifically includes: obtaining a total error of the image in the sliding window according to the error, and optimizing the relative posture of two adjacent frames of images in the sliding window through the total error.

[0013] Based on the same inventive concept, the present invention also proposes an intelligent terminal, which includes a processor and a memory, wherein the processor is communicatively connected to the memory, the memory stores a computer program, and the computer program is used to execute the planar image tracking method as described above.

[0014] Based on the same inventive concept, the present invention further proposes a computer-readable storage medium, wherein the computer-readable storage medium stores program data, and the program data is used to execute the planar image tracking method as described above.

[0015] Compared with the prior art, the beneficial effects of the present invention are: after shooting an image of the target image, the 3D coordinates of the feature points are obtained according to the feature points in the first three frames of the target image, the global pose and relative pose of the subsequently shot images are calculated according to the 3D coordinates, and the global pose is optimized by the error of the relative pose, and the motion tracking of the image is performed according to the optimized global pose, which avoids the need to use a large number of parameters for calculation and the increase in the amount of calculation, has low requirements on hardware and realizes efficient and stable tracking of images, reduces the cost of motion tracking, and expands the scope of use and development direction of motion tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A flow chart of an embodiment of a planar image tracking method of the present invention;

[0017] Figure 2 is a flow chart of another embodiment of the planar image tracking method of the present invention;

[0018] Figure 3 This is a structural diagram of an embodiment of a smart terminal of the present invention;

[0019] Figure 4 This is a structural diagram of an embodiment of a computer-readable storage medium of the present invention. DETAILED DESCRIPTION

[0020] The following describes the implementation of the present application through specific specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the content disclosed in this specification. The present application can also be implemented or applied through other different specific implementations, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the various embodiments of the present disclosure generally described and shown in the accompanying drawings here can be combined with each other without conflict, and the structural components or functional modules therein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the accompanying drawings is not intended to limit the scope of the present disclosure claimed for protection, but merely represents the selected embodiments of the present disclosure. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present disclosure.

[0021] See also Figure 1 to Figure 2 ,in, Figure 1 A flow chart of an embodiment of a planar image tracking method of the present invention; Figure 2 FIG. 1 is a flow chart of another embodiment of the planar image tracking method of the present invention. Figure 1 to Figure 2 The planar image tracking method of the present invention is described in detail.

[0022] In other embodiments, the device executing the planar image tracking method may be a mobile phone, a laptop computer, a desktop computer, a smart phone, smart glasses, and other devices that can be used to calculate relative pose and global pose.

[0023] Among them, the planar image tracking method includes:

[0024] S101: Select a target image, obtain the global pose of the first three frames of the target image and the relative pose between two adjacent frames of the target image according to the correspondence between the feature points of the target image in the first three frames of the target image, and obtain the 3D coordinates of the feature points through the global pose and the adjacent pose.

[0025] In this embodiment, the target image is a plane image, and the step of selecting the target image specifically includes: obtaining the selection conditions of the target image, selecting the target image according to the selection conditions, and extracting and saving the image coordinates and feature vectors of the feature points in the target image. By setting the selection conditions, it is ensured that the target image meets the requirements of feature detection and the final motion tracking effect.

[0026] In one embodiment, the selection conditions are: the target image must have sufficient texture, the area of ​​the pure color (only one color) area cannot be greater than 10% of the total area; the texture should follow a natural distribution, rather than simply repeated; the texture should enrich the entire image, at least reaching a ratio of 90%; the size cannot be too small, and the recommended minimum pixel value is 300. The image that meets the above selection conditions is used as the target image, and the shift feature detection is performed on the target image to extract the feature points and feature vectors of the target image, and the image coordinates and feature vectors of the feature points are saved.

[0027] In this embodiment, the camera on the mobile phone can be used to capture multiple frames of the target image, and motion tracking can be performed based on the images captured by the camera. The mobile phone can also be connected to other cameras, and images can be captured using the connected cameras, and motion tracking can then be performed using the captured images.

[0028] The step of obtaining the global pose of the first three frames of images and the relative pose between two adjacent frames of images based on the correspondence between the feature points of the target image in the first three frames of images of the target image specifically includes: determining the coordinate system where the shooting device is located when shooting the first frame of image as the world coordinate system, obtaining the correspondence between the feature points of the target image in the first three frames of images, and calculating the relative pose between two adjacent frames of images and the global pose of the second frame of image and the third frame of image relative to the world coordinate system based on the correspondence.

[0029] Specifically, the step of obtaining the correspondence between the feature points of the target image in the first three frames of images specifically includes: extracting the feature points and feature vectors in the target image, matching the feature points and feature vectors with the feature points and feature vectors of the target image to calculate the homography matrix, and obtaining the correspondence between the feature points in the first three frames of images according to the homography matrix.

[0030] In this embodiment, after obtaining the global pose and the adjacent pose, a triangulation algorithm is used to calculate the 3D coordinates of the feature points of the target image based on the global pose and the adjacent pose, wherein the 3D coordinates are the coordinates of the feature points in the world coordinate system.

[0031] In a specific embodiment, the coordinate system of the camera corresponding to the first frame image is taken as the world coordinate system, and the images of the first three frames are sequentially subjected to SIFT feature detection, feature points and feature vectors are extracted, and then RANSAC matching is performed with the features of the target image according to the feature points and feature vectors, and the homography matrix is ​​calculated, and the corresponding relationship of the feature points between the three frames is obtained according to these homography matrices, thereby calculating the relative posture between two adjacent frames and their (second frame image, third frame image) posture (global posture) relative to the world coordinate system (the coordinate system of the first frame image). The 3D coordinates of the feature points of the target image are calculated according to the triangulation algorithm.

[0032] Because there is noise in the data of the global pose and the 3D coordinates, the step of obtaining the 3D coordinates of the feature points through the global pose and the adjacent poses also includes: optimizing the global pose and the 3D coordinates through a back-projection error function.

[0033] In this embodiment, the back-projection error function is:

[0034] Among them, K is the camera internal parameter, E belongs to the global pose optimized in the current frame, and u i The image coordinates of the feature point projected to the current frame, X i represents the 3D coordinates of the feature points just calculated, represents the optimized 3D coordinates, SE(3) represents the three-dimensional Euclidean transformation group, R i×j It represents a matrix with i rows and j columns whose elements are real numbers. Since the data of feature points is much smaller than the data of the entire image, the speed and real-time performance of calculation are improved.

[0035] S102: Calculate the global pose of each frame image after the first three frames of images and the relative pose between two adjacent frames of images according to the 3D coordinates, and optimize the global pose and 3D coordinates by a back-projection error function.

[0036] The steps of calculating the global pose of each frame image after the first three frames of images and the relative pose between two adjacent frames of images according to the 3D coordinates specifically include: generating 2D-3D point pairs according to the correspondence between the feature points of the current frame and the first frame and the 3D coordinates of the feature points, calculating the global pose of the current frame through the 2D-3D point pairs, and calculating the relative pose of the current frame and the previous frame through the correspondence between the feature points of the current frame and the previous frame.

[0037] In one embodiment, after obtaining the initial values ​​of the pose and 3D coordinates of the feature points of the first three frames of images, SIFT feature detection is continued for each subsequent frame, and RANSAC feature matching is performed with the target image and the homography matrix is ​​calculated. Similarly, the feature point correspondence between the image pairs is obtained according to the homography matrix. First, 2D-3D point pairs (3D points (feature points) in the first frame and 2D points projected to the current frame) are obtained according to the feature point correspondence between the current frame and the first frame, and the EPnP algorithm is used to calculate its pose relative to the first frame. Then, the essential matrix is ​​calculated according to the feature point correspondence between the current frame and the previous frame, and then the relative pose is obtained by decomposition.

[0038] In this embodiment, the back-projection error function is the same as the back-projection error function used in the previous step: Among them, K is the camera internal parameter, E belongs to the global pose optimized in the current frame, and u i The image coordinates of the feature point projected to the current frame, X iRepresents the 3D coordinates of the feature points (using the values ​​obtained from the last optimization), represents the optimized 3D coordinates, SE(3) represents the three-dimensional Euclidean transformation group, R i×j Represents a matrix with i rows and j columns whose elements are real numbers.

[0039] Among them, after the step of optimizing the global pose and 3D coordinates through the back projection error function, it also includes: putting the current frame into the sliding window, and when the number of frames in the sliding window is greater than a preset value, removing the frame image with the smallest sequence number in the sliding window.

[0040] In this embodiment, the size of the sliding window is 6 frames, the step size is one frame, and the preset value is 6. In other embodiments, the size and step size of the sliding window may not be limited to the above values, and the specific size may be set according to the actual environment and requirements.

[0041] S103: Calculate the relative pose respectively through the feature points and global pose of two adjacent frames of images, optimize the relative pose according to the error of the relative pose, calculate the global pose of the image through the optimized relative pose, and use the global pose to perform real-time tracking and pose optimization for each frame of image.

[0042] The relative pose is calculated through the correspondence between the feature points in two adjacent frames of images, and the global pose of the two adjacent frames of images is obtained. The relative pose between the two adjacent frames of images is calculated using the global pose, and the error between the two calculated relative poses is obtained.

[0043] In this embodiment, the step of optimizing the relative posture according to the error of the relative posture specifically includes: obtaining the total error of the image in the sliding window according to the error, and optimizing the relative posture of two adjacent frames of images in the sliding window through the total error.

[0044] In one embodiment, the relative pose calculated based on the feature points between adjacent frames is recorded as The relative pose derived from the global pose of adjacent frames is denoted as T ij Now define the error e ij (represents the relative posture T calculated by the formula ij And the relative pose calculated by feature points The degree of deviation) is:

[0045] The total error is Where N represents the adjacent frames in the sliding window, T i represents the global pose of the i-th frame, T jrepresents the global pose of the jth frame, SE(3) represents the three-dimensional Euclidean transformation group, and then the g2o open source library in C++ is used to optimize (3) to obtain the optimal relative pose. According to the optimal relative pose, the optimal global pose of each frame in the sliding window is obtained, and this cycle is continued to perform real-time motion tracking and pose optimization (obtain the 3D coordinates of the optimized feature points) for each subsequent frame.

[0046] Beneficial effect: After the planar image tracking method of the present invention captures an image of the target image, the 3D coordinates of the feature points in the first three frames of the target image are obtained, and the global pose and relative pose of the subsequently captured images are calculated based on the 3D coordinates. The global pose is optimized by the error of the relative pose, and the motion tracking of the image is performed based on the optimized global pose, which avoids the need to use a large number of parameters for calculation and the increase in the amount of calculation. It has low hardware requirements and achieves efficient and stable tracking of images, reduces the cost of motion tracking, and expands the scope of use and development direction of motion tracking.

[0047] Based on the same inventive concept, the present invention also proposes an intelligent terminal, see Figure 3 , Figure 3 This is a structural diagram of an embodiment of the intelligent terminal of the present invention, combined with Figure 3 The intelligent terminal of the present invention is described.

[0048] In this embodiment, the intelligent terminal includes a processor and a memory. The processor is communicatively connected to the memory. The memory stores a computer program. The computer program is used to execute the planar image tracking method described in the above embodiment.

[0049] In some embodiments, the memory may include, but is not limited to, high-speed random access memory, non-volatile memory. For example, one or more disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable functional devices, discrete gates or transistor functional devices, discrete hardware components.

[0050] Based on the same inventive concept, the present invention also provides a computer-readable storage medium, see Figure 4 , Figure 4 4 is a structural diagram of an embodiment of a computer-readable storage medium of the present invention, and the computer-readable storage medium of the present invention is described in conjunction with FIG.

[0051] In this embodiment, a computer-readable storage medium stores program data, and the program data is used to execute the planar image tracking method described in the above embodiment.

[0052] The computer-readable storage medium may include, but is not limited to, a floppy disk, an optical disk, a CD-ROM (compact disk-read only memory), a magneto-optical disk, a ROM (read only memory), a RAM (random access memory), an EPROM (erasable programmable read-only memory), an EEPROM (electrically erasable programmable read-only memory), a magnetic card or an optical card, a flash memory, or other types of media / machine-readable media suitable for storing machine-executable instructions. The computer-readable storage medium may be a product that is not connected to a computer device, or a component that is connected to a computer device for use.

[0053] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0054] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A planar image tracking method, characterized in that: The planar image tracking method comprises: Select a target image, obtain the global pose of the first three frames of images and the relative pose between two adjacent frames of images according to the correspondence between the feature points of the target image in the first three frames of images of the target image, and obtain the 3D coordinates of the feature points through the global pose and the adjacent poses, wherein the coordinate system in which the shooting device is located when shooting the first frame of image is determined as the world coordinate system, obtain the correspondence between the feature points of the target image in the first three frames of images, and calculate the relative pose between two adjacent frames of images and the global pose of the second frame of image and the third frame of image relative to the world coordinate system according to the correspondence; The global pose of each frame image after the first three frames of images and the relative pose between two adjacent frames of images are calculated according to the 3D coordinates, and the global pose and 3D coordinates are optimized by a back-projection error function, wherein the back-projection error function is: in, is the camera internal parameter, The global pose that belongs to the optimization of the current frame, The image coordinates of the feature points projected to the current frame, represents the 3D coordinates of the feature points just calculated, represents the optimized 3D coordinates, SE(3) represents the three-dimensional Euclidean transformation group, Represents elements as real numbers OK Column matrix; Relative poses are calculated respectively by using feature points and global poses of two adjacent frames of images, and the relative poses are optimized according to the errors of the relative poses, including: placing the current frame into a sliding window, and when the number of frames in the sliding window is greater than a preset value, removing a frame of image with the smallest sequence number in the sliding window; obtaining a total error of the image in the sliding window according to the error, optimizing the relative poses of two adjacent frames of image in the sliding window according to the total error, calculating the global pose of the image according to the optimized relative pose, and using the global pose to perform real-time tracking and pose optimization on each frame of image.

2. The planar image tracking method according to claim 1, characterized in that: The step of selecting the target image specifically includes: A selection condition of a target image is obtained, a target image is selected according to the selection condition, and image coordinates and feature vectors of feature points in the target image are extracted and saved.

3. The planar image tracking method according to claim 1, wherein: The step of obtaining the correspondence between the feature points of the target image in the first three frames of images specifically includes: Extract feature points and feature vectors in the target image, match the feature points and feature vectors with the feature points and feature vectors of the target image to calculate a homography matrix, and obtain the correspondence between the feature points in the first three frames of images according to the homography matrix.

4. The planar image tracking method according to claim 1, wherein: The step of calculating the global pose of each frame image after the first three frames image and the relative pose between two adjacent frames image according to the 3D coordinates specifically includes: A 2D-3D point pair is generated according to the correspondence between the feature points of the current frame and the first frame and the 3D coordinates of the feature points. The global pose of the current frame is calculated through the 2D-3D point pair, and the relative pose of the current frame and the previous frame is calculated through the correspondence between the feature points of the current frame and the previous frame.

5. An intelligent terminal, characterized in that: The intelligent terminal includes a processor and a memory, wherein the processor is in communication with the memory, and the memory stores a computer program, and the computer program is used to execute the planar image tracking method according to any one of claims 1 to 4.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program data, and the program data is used to execute the planar image tracking method according to any one of claims 1 to 4.

Images