Image area feature and visual text feature multi-feature fusion tracking method

Abstract

The embodiment of the invention provides an image region feature and visual text feature multi-feature fusion tracking method, which can solve the technical problem of low tracking precision of a symmetric object with a text feature. The method comprises the following steps: giving first pose information of target equipment; matching a target viewpoint template drawing according to the first pose information; the target viewpoint template graph is a template graph in the sparse viewpoint model; the sparse viewpoint model is generated on the basis of different postures of the target equipment; based on the first image frame of the target device and the target viewpoint template graph, determining second pose information of the target device through regional text texture tracking; and determining a tracking image result of the target equipment based on the second pose information. According to the invention, the tracking precision of the industrial equipment of the symmetric object with text features is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a multi-feature fusion tracking method of image region features and visual text features. Background Art

[0002] Industrial equipment is often a symmetrical object with regular shapes and features letters and Chinese characters in standardized fonts. The commonly used single tracking method results in low tracking accuracy and is not well adapted to the tracking of industrial equipment. Summary of the invention

[0003] The embodiment of the present application provides a multi-feature fusion tracking method of image region features and visual text features, which can solve the technical problem of low tracking accuracy of symmetrical objects with text features.

[0004] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0005] In the first aspect, an embodiment of the present application provides a multi-feature fusion tracking method of image region features and visual text features, and the multi-feature fusion tracking method of image region features and visual text features includes: given the first pose information of a target device; according to the first pose information, matching the target viewpoint template map; the target viewpoint template map is a template map in a sparse viewpoint model; the sparse viewpoint model is generated based on the target device in different poses; based on the first image frame of the target device and the target viewpoint template map, confirming the second pose information of the target device through regional texture tracking; based on the second pose information, confirming the tracking image result of the target device.

[0006] Based on the above description of the image region feature and visual text feature multi-feature fusion tracking method provided in the embodiment of the present application, it can be seen that the image region feature and visual text feature multi-feature fusion tracking method is combined with the relatively symmetrical appearance structure of the tracking product, which easily leads to poor tracking effect. A tracking technology based on image multi-feature fusion is proposed. And, based on the real equipment of the real space station, relevant experiments are used to show the actual effectiveness.

[0007] In a feasible implementation of the first aspect, when executing the step of matching the target viewpoint template map according to the first pose information, the multi-feature fusion tracking method of image area features and visual text features also includes: setting different viewpoint poses collected by the target device template; generating multiple pose projection maps based on the viewpoint pose; calculating the first vector corresponding to each pose projection map, the first vector being the vector of the virtual camera pointing to the center of the device model; and calculating the target viewpoint template map based on the first vector.

[0008] In a feasible implementation of the first aspect, the multi-feature fusion tracking method of image region features and visual text features also includes: performing regional texture tracking based on three-dimensional information; wherein the three-dimensional information includes the three-dimensional coordinates and three-dimensional normal vectors of the contour sampling points.

[0009] In a feasible implementation of the first aspect, when executing the step of confirming the second pose information of the target device through regional texture tracking based on the first image frame and the target viewpoint template map of the target device, the multi-feature fusion tracking method of image region features and visual text features also includes: obtaining the estimated probability of a specific pose of multiple modal data of the camera based on the probability density function of regional texture tracking; and maximizing the estimated probability based on the maximization function to confirm the second pose information.

[0010] In a feasible implementation of the first aspect, when executing the steps of maximizing the estimated probability based on the maximization function and confirming the second posture information, the multi-feature fusion tracking method of image area features and visual text features also includes: based on global optimization or local optimization, optimizing the first-order partial derivatives and second-order partial derivatives of the natural logarithm of the probability density function to obtain optimized first-order partial derivatives and optimized second-order partial derivatives; estimating posture variables based on the optimized first-order partial derivatives and optimized second-order partial derivatives.

[0011] In a feasible implementation of the first aspect, the grayscale difference between at least nine consecutive pixels and the center point pixel is greater than a threshold, and the remaining pixels and the center point pixel meet a similar threshold.

[0012] In a feasible implementation of the first aspect, the calculation formula for regional texture tracking includes:

[0013]

[0014] Among them, p(θ|D r ) is expressed as the regional characteristic probability density function; p(θ|D t ) is expressed as the text feature probability density function; n t The number of contour sampling points represented as regional features; n r Represented as the number of text feature points.

[0015] In a feasible implementation of the first aspect,

[0016]

[0017] Where n represents the number of text feature points; ρ tuk Expressed as a function that minimizes the impact of outliers; σ t Expressed as the regularized variance coefficient; x i Indicates the key points detected in the current frame; x iIndicates the key point in the key frame; n c Expressed as the number of contour sampling points; d si ,ω si and l si It is expressed as the value of the discrete point of the corresponding line; θ is expressed as the change in posture.

[0018] In a second aspect, an embodiment of the present application provides a multi-feature fusion tracking system of image region features and visual text features, wherein the multi-feature fusion tracking system of image region features and visual text features comprises: at least one processor; a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method provided in the first aspect.

[0019] In a third aspect, an embodiment of the present application provides a computer-readable medium having computer program instructions stored thereon, and the computer program instructions can be executed by a processor to implement the method provided in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A schematic diagram of the structure of a multi-feature fusion tracking system of image region features and visual text features provided in an embodiment of the present application;

[0021] Figure 2 A flowchart of a multi-feature fusion tracking method of image region features and visual text features provided in an embodiment of the present application;

[0022] Figure 3 A sparse viewpoint partitioning diagram in a multi-feature fusion tracking method of image region features and visual text features provided in an embodiment of the present application;

[0023] Figure 4 A schematic diagram of contour sampling points in a multi-feature fusion tracking method of image region features and visual text features provided in an embodiment of the present application;

[0024] Figure 5 A schematic diagram of corresponding lines in a multi-feature fusion tracking method of image region features and visual text features provided in an embodiment of the present application;

[0025] Figure 6 A schematic diagram of feature point selection in a multi-feature fusion tracking method of image region features and visual text features provided in an embodiment of the present application;

[0026] Figure 7 A schematic diagram of feature point selection in a multi-feature fusion tracking method of image region features and visual text features provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] The technical solutions in the embodiments of the present invention will be described below in conjunction with the accompanying drawings in the embodiments of the present invention. In the description of the embodiments of the present invention, unless otherwise specified, "multiple" refers to two or more than two. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0028] In addition, in order to facilitate the clear description of the technical solutions of the embodiments of the present invention, in the embodiments of the present invention, the words "first", "second" and the like are used to distinguish the same items or similar items with substantially the same functions and effects. Those skilled in the art will understand that the words "first", "second" and the like do not limit the quantity and execution order, and the words "first", "second" and the like do not necessarily limit the difference. At the same time, in the embodiments of the present invention, the words "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or design. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete way for easy understanding.

[0029] The principles and features of the present application are described below. The examples given are only used to explain the present application and are not used to limit the scope of the present application.

[0030] Tracking technology is the core technology of augmented reality technology to achieve virtual and real registration, and it is also the most important technology. How to accurately and robustly track so as to accurately superimpose virtual guidance information at the corresponding position and posture is an urgent problem to be solved.

[0031] First of all, the virtual-to-real registration method based on manual identification requires posting artificially designed black-and-white identifiers with obvious characteristics in the real maintenance environment, which will not only make the environment unsightly and cause discomfort to operators; at the same time, not all equipment is suitable for posting artificial identifiers, and the size of the identifier will also affect the accuracy of the virtual-to-real registration, and it is extremely sensitive to the occlusion of the identifier.

[0032] Secondly, spatial products are basically devices with weak texture or no texture, which leads to the fact that the virtual-real registration method based on feature points often cannot achieve ideal results.

[0033] Then, the shapes of spatial products are mostly similar geometric structures, which leads to a large number of similar edges in the environment, which makes the edge-based virtual-real registration method easily fail in cluttered backgrounds.

[0034] In addition, deep learning-based methods are currently developing rapidly, but these methods rely on large-scale data, long training times, and high computing resources. However, the data on our space station equipment is limited and difficult to obtain, and we hope to obtain real-time calculations.

[0035] The embodiment of the present application provides a tracking method that integrates image region features and visual text features, which is suitable for space equipment tracking to achieve virtual-real registration. The appearance structure of space products is relatively symmetrical, which easily leads to poor virtual-real registration effect. A virtual-real registration technology based on image multi-feature fusion is proposed.

[0036] In some embodiments, in the offline stage, a sparse viewpoint model of the device is established and target viewpoint templates in different postures are collected for matching in the online stage. In the online stage, a video stream of a real maintenance scene is captured by a camera. The pose information of the device is obtained through point cloud recognition technology, and the viewpoint template that best matches the viewpoint model template is found. Joint optimization is performed based on region and texture features to improve the accuracy and robustness of pose estimation.

[0037] The embodiment of the present application provides a multi-feature fusion tracking system of image region features and visual text features, which can execute the multi-feature fusion tracking method of image region features and visual text features provided by the embodiment of the present application. Figure 1 A schematic diagram of the structure of a multi-feature fusion tracking system of image region features and visual text features provided in an embodiment of the present application.

[0038] like Figure 1 As shown, the image region feature and visual text feature multi-feature fusion tracking system 001 includes at least one processor 011 and a memory 012 communicatively connected to the at least one processor; wherein the memory 012 stores instructions executable by the at least one processor 011, and the instructions are processed by the at least one region and text multi-feature fusion tracking method.

[0039] Figure 2 A flowchart of a multi-feature fusion tracking method of image region features and visual text features provided in an embodiment of the present application. Figure 2 As shown, in some embodiments, the image region feature and visual text feature multi-feature fusion tracking method includes the following steps:

[0040] S1, the first pose information of the given target device.

[0041] The first pose information is the relative position and pose transformation of the object relative to the camera. The first pose information includes the rotation transformation (matrix) R and the translation transformation t.

[0042] S2, match the target viewpoint template image according to the first pose information.

[0043] The target viewpoint template graph is a template graph in the sparse viewpoint model.

[0044] The sparse viewpoint model is generated based on the target device in different postures. From the perspective of the virtual camera, the three-dimensional vector values ​​of the contour sampling points of the target device in different postures, the normal vectors of the contour sampling points, and the vector of the virtual camera pointing to the center of the model are all stored in the sparse viewpoint model. In this way, the sparse viewpoint model can efficiently represent the target device through the model contour.

[0045] In some embodiments, according to the first pose information, when matching the target viewpoint template map, it can be simplified to search for the closest pre-calculated view. In one implementation, when executing step S2, the image region feature and visual text feature multi-feature fusion tracking method also includes the following steps:

[0046] S21, setting different viewpoint poses for target device template collection.

[0047] In some embodiments, a 3D geometric shape is rendered from multiple different perspectives around the target device, a virtual camera is placed at a vertex of a grid surrounding the target device, and the target device is set to be placed in the center of the grid. Viewpoint template images of the target device in different positions under the perspective of the virtual camera are collected. Figure 3 A sparse viewpoint partitioning diagram in a multi-feature fusion tracking method of image region features and visual text features provided in an embodiment of the present application. Figure 3 As shown, the sparse viewpoint division process is achieved by subdividing each triangle of the regular icosahedron multiple times. The “multiple subdivisions” can be as follows Figure 3 The three times shown may also be four times. The number of "multiple subdivisions" is not limited in this application.

[0048] In some embodiments, the posture projection map is generated from 2562 different angles. The posture projection map can be as follows: Figure 3 The circular scatter plot shown.

[0049] S22, generating multiple posture projection images based on the viewpoint posture.

[0050] In some embodiments, the pose projection map is obtained according to a sparse viewpoint method.

[0051] By executing step S21 and step S22, a sparse viewpoint model is obtained.

[0052] S23, calculating the first vector corresponding to each posture projection graph.

[0053] The first vector is the vector that the virtual camera points to the center of the device model.

[0054] In some embodiments, the calculation formula of the vector pointing from the virtual camera to the center of the sparse viewpoint model includes:

[0055] M v= M R C e Z ;

[0056] in, M v is represented by the vector pointing from the camera to the center of the object model, e Z =[0 0 1] T Represented as a unit vector.

[0057] S24, calculating a target viewpoint template map based on the first vector.

[0058] In some embodiments, the calculation formula of the target viewpoint template graph includes:

[0059]

[0060] By executing steps S21 to S24, the vector pointing from the camera to the center of the device model, ie, the first vector, is stored.

[0061] S3, based on the first image frame of the target device and the target viewpoint template map, confirm the second posture information of the target device through regional texture tracking.

[0062] In some embodiments, when executing step S3, the image region feature and visual text feature multi-feature fusion tracking method further includes the following steps:

[0063] S31, based on the probability density function of regional texture tracking, obtain the estimated probability of a specific posture of multiple modal data of the camera.

[0064] In some embodiments, the calculation formula of the probability density function includes:

[0065]

[0066] The following explains p(θ|D t ) and p(θ|D r ).

[0067] First, we explain p(θ|D t ):

[0068]

[0069] in,

[0070] x i(θ)=π( C T M M T(θ) M X i );

[0071]

[0072] Where n represents the number of text feature points; ρ tuk Expressed as a function that minimizes the impact of outliers; σ t Expressed as the regularized variance coefficient; x i Indicates the key points detected in the current frame; x i Represents a key point in a keyframe.

[0073] like Figure 4 As shown, the surface of the target device also has texture features, such as the word "filter". In some embodiments, the texture of the outer surface of the space station device is mainly composed of obvious texture features such as text and characters.

[0074] Then, we explain p(θ|D r ):

[0075] Based on region tracking, the developed sparse viewpoint model is combined with the corresponding line model to define the probability density function of posture change. However, before calculating the probability, the position and direction of the corresponding line need to be defined. Among them, some three-dimensional information is involved in region tracking. The three-dimensional information includes the three-dimensional coordinates of the contour sampling points and the three-dimensional normal vector.

[0076] The calculation formulas for the three-dimensional information include:

[0077]

[0078] Among them, x i Represented as two-dimensional information of contour sampling points; Represented as rendering depth; It is expressed as the inverse process of the camera imaging model and returns to the homogeneous form; n i is represented as the normal vector of the corresponding point, and ||n i ||2=1; M T C Represented as the relative pose transformation matrix from the camera reference frame C to the model reference frame M; M R C Expressed as M T C The corresponding rotation matrix; The three-dimensional information of the three-dimensional point corresponding to the two-dimensional point; M N i Represented as the three-dimensional vector value corresponding to the two-dimensional vector.

[0079] According to the three-dimensional information, the two-dimensional information of the corresponding line is obtained.

[0080]

[0081] n i ∝( c R M M N i ) 2×1 ;

[0082] in,() 2×1 Represents the first two corresponding elements of a vector.

[0083] Then, define the corresponding line:

[0084]

[0085] Among them, d i (θ) is the contour distance relative to the posture change vector θ; the contour distance is along the normal vector n i From the center of the corresponding line c i To project 3D model point X i The distance; c i is the i-th sampling point on the model projection contour; n i Indicates point c i Normal of i is the c in the model i 3D model point; π() is the projection function of the pinhole camera that projects a three-dimensional space point onto a two-dimensional image.

[0086] Then, we assume that multiple corresponding lines are independent and use the discrete scale space to calculate p(θ|D r ) probability density function.

[0087] In order to better detect text features, in some embodiments, such as Figure 6 As shown in the figure, the left side represents the stroke ending (SEK) feature point, which focuses on the detection of the stroke endpoint of the texture feature. The judgment condition is that the grayscale difference between at least nine consecutive pixels and the center point pixel must be greater than the threshold, and the remaining pixels are close to the center point pixel. Figure 7 As shown, the middle right side represents the stroke bend keypoint (SBK) feature point, which focuses on the extraction of feature points at the turning point of the stroke. The judgment condition is that the grayscale difference between at least six consecutive pixels and the center pixel must be greater than the threshold, or there must be two different pixels that are similar points, and the other points are bright points or dark points.

[0088] A feasible implementation method for determining bright spots, dark spots, and similar spots is given below.

[0089] In some embodiments, the pixels on the annular band are divided into three categories: darker (i.e., dark points), similar (i.e., similar points), and brighter (i.e., bright points) according to the difference between them and the center point pixel, and according to the difference from small to large:

[0090]

[0091] Among them, p represents the pixel point in the image, and its corresponding pixel value is I p , which is the threshold m of the 12 pixel points x and p on the circle of the two pixel distances. According to the size of the difference, different pixel points can be classified into three categories, among which P d , P s , P b It is a collection of three types of pixels: darker, similar, and brighter.

[0092] In order to eliminate key points that are close to each other, a simple non-maximum suppression is performed on the 3×3 neighborhood. Only the key points with the highest contrast (i.e., max(I p -I p ): x∈P b max(I p -I p ): x∈P d ).

[0093] Individual characters can be segmented from the background using a threshold unique to each character, which can be found directly from the keypoints. Foreground is black pixels. Given a keypoint p and its associated set of darker pixels P d , segmentation threshold θ p Higher than P d The intensity value of the darkest pixel intensity, that is:

[0094] θ p =max(I X )+1|x∈P d .

[0095] When the foreground is a white pixel, the segmentation threshold is the set of pixels P that are darker than the center pixel. b The minimum pixel value in , that is:

[0096] θ p =min(I X )-1|x∈P b .

[0097] The embodiment of the present application has a good effect not only on the text features with inflection points or the endpoints of the text, but also on the extraction of text features without corners and endpoints such as the number '0'.

[0098] In some embodiments, the point-to-point distance of the posture corresponding to the texture feature is obtained by minimization.

[0099] In one implementation, the method based on text texture corresponding to the second pose information aims to minimize the distance between corresponding feature points. The second pose is the pose corresponding to the second pose information. New key points are detected in a rectangular area adjacent to the first pose of the first image frame (i.e., the area obtained by projecting the first pose on the image) as pose key points (a matrix of a preset size). The descriptors of the pose key points are matched with the features of the first key frame. If the direction difference with the first key frame exceeds a specific threshold, a second key frame is established. A depth rendering of the second key frame is generated, and a three-dimensional (3D) model point is reconstructed for each pose key point (which can be regarded as a contour sampling point) that falls on the contour of the second key frame. Among them, in addition to the descriptors of the contour sampling points of the second key frame, unobstructed 3D points are also stored in the key frame.

[0100] The first posture may be a given posture. The target viewpoint template image is related to the first posture. In this way, the 3D model points and normal vectors in the target viewpoint template image are projected onto the first image frame.

[0101] It is understandable that the first key frame includes an image frame corresponding to the first posture. In one implementation, the first key frame is the first image frame.

[0102] This helps reduce drift and enhances the robustness of the overall tracking by enabling integration of keyframes.

[0103] Next, we explain p(θ|D r ):

[0104]

[0105] Among them, D r Represented as data of all corresponding lines; n c Expressed as the number of contour sampling points; d si ,ω si and l si It is expressed as the value of the discrete point of the corresponding line; θ is expressed as the change in posture.

[0106] The "corresponding line" is explained in detail below.

[0107] The region-based method of the related art needs to calculate the entire region, while the sparse region-based method provided by the present application only needs to calculate the elements on the corresponding straight line, that is, Figure 5 The elements on the “corresponding line” are shown. The sparse region based method can not only achieve higher accuracy but also significantly improve the computational efficiency.

[0108] In some embodiments, the corresponding line is defined as:

[0109] l(r)=I(c+rn);

[0110] Among them, c represents the center of the corresponding line, represents the normal vector, and ||n||2=1; I represents the image; r represents the line coordinate; and the contour distance d represents the distance from the center of the corresponding line to the estimated contour.

[0111] S32, based on the maximization function, maximize the estimated probability and confirm the second posture information.

[0112] In some embodiments, Gauss-Newton optimization and Tikhonov regularization are used as a method to maximize the function to maximize p(θ|D) to estimate the pose variable Then confirm the second posture information.

[0113] Posture variables The calculation formula includes:

[0114]

[0115] Among them, the parameter λ r and λ t are respectively represented as the regularization terms of rotation and translation; the gradient g and Hessian matrix H are respectively represented as the first-order partial derivative and second-order partial derivative of the natural logarithm of the joint probability density function.

[0116] First-order partial derivative, gradient g r The transpose of g r T The calculation formula includes:

[0117]

[0118] in,

[0119] The calculation formula of the second-order partial derivative, Hessian matrix H, includes:

[0120]

[0121] g is expressed as the first-order partial derivative of the regional text multi-feature probability density function, and the calculation formula includes:

[0122] g=g r +g t ;

[0123] Among them, g t Expressed as the first-order partial derivative of the text feature probability density function, g t The transpose of g t T The calculation formula includes:

[0124]

[0125] H represents the second-order partial derivative of the regional text multi-feature probability density function, and the calculation formula includes:

[0126] H=H r +H t ;

[0127] Among them, H t It is expressed as the second-order partial derivative of the text feature probability density function. The calculation formula includes:

[0128]

[0129] By repeatedly repeating step S321 , a posture that maximizes the joint probability density function combining text texture and region information can be obtained.

[0130] S322, estimating the posture variable based on the optimized first-order partial derivative and the optimized second-order partial derivative.

[0131] In this way, the second pose information of the target device is confirmed by using p(θ|D) to represent the likelihood of a specific pose of the target device given the multimodal data, and then estimating the pose by maximizing the function.

[0132] S4, confirming the tracking image result of the target device based on the second posture information.

[0133] Based on the same application concept, an image region feature and visual text feature multi-feature fusion tracking system is also provided in an embodiment of the present application. The method corresponding to the image region feature and visual text feature multi-feature fusion tracking system may be the image region feature and visual text feature multi-feature fusion tracking method in the aforementioned embodiment, and its principle of solving the problem is similar to that of the method. The image region feature and visual text feature multi-feature fusion tracking system provided in an embodiment of the present application includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the methods and / or technical solutions of the aforementioned multiple embodiments of the present application.

[0134] Another embodiment of the present application further provides a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions can be executed by a processor to implement the methods and / or technical solutions of any one or more embodiments of the present application described above.

[0135] Specifically, the present embodiment may adopt any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device, or device.

[0136] Computer readable signal media may include a data signal propagated in baseband or as part of a carrier wave, which carries a computer readable program code. Such propagated data signals may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than a computer readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0137] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0138] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0139] The flow chart or block diagram in the accompanying drawings shows the possible architecture, function and operation of the equipment, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated system for hardware that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0140] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0141] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or page components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0142] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0143] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0144] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.

[0145] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

[0146] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in a device claim can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any particular order.

Claims

1. A multi-feature fusion tracking method of image region features and visual text features, characterized in that: include: Given the first pose information of the target device; According to the first posture information, a target viewpoint template image is matched; the target viewpoint template image is a template image in a sparse viewpoint model; The sparse viewpoint model is generated based on the target device in different postures; Based on the first image frame of the target device and the target viewpoint template map, confirming the second pose information of the target device through regional texture tracking; Based on the second posture information, a tracking image result of the target device is confirmed.

2. The image region feature and visual text feature multi-feature fusion tracking method according to claim 1 is characterized in that: When performing the step of matching the target viewpoint template map according to the first pose information, the image region feature and visual text feature multi-feature fusion tracking method further includes: Setting different viewpoint poses collected by the target device template; Based on the viewpoint posture, generating a plurality of posture projection images; Calculate a first vector corresponding to each of the posture projection images, where the first vector is a vector that the virtual camera points to the center of the device model; Based on the first vector, the target viewpoint template map is calculated.

3. The image region feature and visual text feature multi-feature fusion tracking method according to claim 2 is characterized in that: The image region feature and visual text feature multi-feature fusion tracking method also includes: Based on the three-dimensional information, the regional texture tracking is performed; wherein, The three-dimensional information includes three-dimensional coordinates and three-dimensional normal vectors of contour sampling points.

4. The image region feature and visual text feature multi-feature fusion tracking method according to any one of claims 1 to 3, characterized in that: When executing the step of confirming the second pose information of the target device by regional texture tracking based on the first image frame of the target device and the target viewpoint template map, the image region feature and visual text feature multi-feature fusion tracking method further includes: Based on the probability density function of the regional texture tracking, obtaining an estimated probability of a specific posture of multiple modal data of the camera; Based on a maximization function, the estimated probability is maximized and the second posture information is confirmed.

5. The image region feature and visual text feature multi-feature fusion tracking method according to claim 4 is characterized in that: When executing the step of maximizing the estimated probability based on the maximization function and confirming the second pose information, the image region feature and visual text feature multi-feature fusion tracking method further includes: Based on global optimization or local optimization, optimizing the first-order partial derivative and the second-order partial derivative of the natural logarithm of the probability density function to obtain an optimized first-order partial derivative and an optimized second-order partial derivative; The posture variables are estimated based on the optimized first-order partial derivatives and the optimized second-order partial derivatives.

6. The image region feature and visual text feature multi-feature fusion tracking method according to any one of claims 1 to 3, characterized in that: The image region feature and visual text feature multi-feature fusion tracking method also includes: At the same time, the grayscale difference between at least nine consecutive pixels and the center point pixel is greater than the threshold, and the remaining pixels and the center point pixel meet the similar threshold.

7. The image region feature and visual text feature multi-feature fusion tracking method according to any one of claims 1 to 3, characterized in that: The calculation formula for regional texture tracking includes: Among them, p(θ|D r ) is expressed as the regional characteristic probability density function; p(θ|D t ) is expressed as the text feature probability density function; n t The number of contour sampling points represented as regional features; n r Represented as the number of text feature points.

8. The image region and text feature fusion tracking method according to claim 7, characterized in that: Where n is the number of text feature points; ρtuk is the function that minimizes the impact of outliers; σ t Expressed as the regularized variance coefficient; x i Indicates the key points detected in the current frame; x i Indicates the key point in the key frame; n c Expressed as the number of contour sampling points; d si ,ω si and l si It is expressed as the value of the discrete point of the corresponding line; θ is expressed as the change in posture.

9. A multi-feature fusion tracking system of image region features and visual text features, characterized in that: include: at least one processor; a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method according to any one of claims 1 to 8.

10. A computer readable medium having computer program instructions stored thereon, wherein the computer program instructions can be executed by a processor to implement the method according to any one of claims 1 to 8.

Images