A virtual-real synchronization method based on mixed reality technology

Abstract

The present application relates to the field of information processing, comprising a kind of virtual-real synchronization method based on mixed reality technology, first first algorithm initialization is carried out, and initial distribution is obtained using feature point matching and first frame image, then through pose estimation value precision determination basis selection stable feature point, in turn carry out optical flow method tracking, three-dimensional reconstruction and ukf position estimation;Then through position estimation criterion-reprojection error, determine whether to add to the three-dimensional point sequence of pose estimation;Finally, the algorithm is based on three-dimensional point and carries out pose estimation, when scene point moves out in the process of algorithm operation, then delete from the three-dimensional point sequence of current pose estimation, when there is new feature point, three-dimensional reconstruction is carried out again, stable new point is put into three-dimensional point sequence after position estimation and carries out pose estimation.The present application can solve the problem of limited view, and can cope with the situation such as occlusion camera jitter illumination and viewpoint, angle of view change, can satisfy indoor and outdoor scene application.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of mixed reality technology, in particular to a virtual-real synchronization method based on mixed reality technology. BACKGROUND

[0002] Mixed reality is a combination of technologies that not only provides new viewing methods, but also provides new input methods, and all methods are combined with each other to promote innovation. The combination of input and output is a key differentiating advantage for small and medium-sized enterprises. In this way, mixed reality can directly affect your workflow and help you and your employees improve work efficiency and innovation. Mixed reality technology (MR) is a further development of virtual reality technology, which presents virtual scene information in a real scene, and builds an interactive feedback information loop between the real world, the virtual world and the user to enhance the realism of the user experience. The traditional mixed reality is limited by the use of the scene. For example, indoor use cannot be used outdoors, or the method is limited in the field of view after being trained, so a virtual-real synchronization method based on mixed reality technology is needed. SUMMARY

[0003] The purpose of the present application is to provide a virtual-real synchronization method based on mixed reality technology;

[0004] The present application is implemented in the following way, the present application provides a virtual-real synchronization method based on mixed reality technology; the specific steps are as follows,

[0005] S1: First, the algorithm is initialized, and the initial distribution is obtained by using feature point matching and the first frame image, then the feature points with stable comparison are selected by the pose estimation value accuracy judgment basis, and the light flow tracking, three-dimensional reconstruction and ukf position estimation are sequentially performed;

[0006] S2: Then, whether to add to the three-dimensional point sequence of the pose estimation is judged by the position estimation criterion-reprojection error;

[0007] S3: Finally, the algorithm estimates the pose according to the three-dimensional points, and when the scene points move out during the algorithm running process, the three-dimensional point sequence of the current pose estimation is deleted, and when there are new feature points, the three-dimensional reconstruction is performed again, and the stable new points are put into the three-dimensional point sequence for pose estimation after position estimation; then the center of gravity and search radius of the M feature point particle re-projection distribution are used to obtain the candidate measurement area corresponding to the M feature points.

[0008] S4: Steps S1-S3 are repeated.

[0009] Further, a simple and fast feature point extraction method-Fast is adopted to avoid time-consuming Gaussian difference filtering on the image to improve the real-time performance of the system.

[0010] Further, the reference image is subjected to scale and affine transformation through the offline stage to obtain a descriptor set of the same feature point under different viewpoints and angles, so as to realize the robustness of the system to the viewpoint and angle changes of the user.

[0011] Further, the feature point matching strategy of the simplified SIFT algorithm is based on the scale transformation of the reference image by using the Gaussian filter, and then the appearance of the feature point under the viewpoint and angle changes is simulated by using the affine transformation shown in formula (1) to generate a descriptor set.

[0012] Formula (1)

[0013] Wherein, x0 is the position of the feature point on the reference image, x'0 is the center of the transformed image, and x' is the position of x after transformation.

[0014] Further, in step S1, the optical flow method tracking, three-dimensional reconstruction and ukf position estimation are sequentially performed, a reference frame and a reference system of the feature point are first selected, then the position of the feature point in the corresponding reference frame is determined, and the three-dimensional point position of the corresponding image feature is estimated, so that the map construction can be divided into three main steps: selecting a reference frame, calculating the three-dimensional position of the feature point in the reference frame and map update control.

[0015] Further, in step S3, the measurement process includes the projection of the currently estimated three-dimensional scene point to the estimated mean k of the camera pose, and the estimated value is derived from the particle filter, first, the projection of the feature point zm in the camera state, specifically as formula (2);

[0016] Formula (2)

[0017] Wherein, Π refers to pinhole imaging projection, so that the process of performing measurement only uses xk in the particle filter at k moment, that is, the stability of the reprojected point is determined by the re-projection error constraint.

[0018] Further, the application provides a computer readable storage medium, which stores a computer program, and the program is executed by a host controller to realize the method in any one of the above embodiments. Compared with the prior art, the application has the following beneficial effects:

[0019] The application realizes the pose calculation of the target in the whole scene by using the apf for pose estimation and the ukf for position estimation, and finally completes the virtual-real registration. The method can solve the limited view problem, and can cope with the situations such as occlusion, camera jitter, light, and viewpoint and angle changes, and can meet the indoor and outdoor scene applications. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some of the embodiments of the present application, and therefore should not be regarded as a limitation to the scope. For those skilled in the art, other related drawings can also be obtained without creative labor.

[0021] Figure 1 is a schematic diagram of the method process of the present application. DETAILED DESCRIPTION

[0022] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the following will combine the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0023] Please refer to Figure 1 A virtual-real synchronous method based on mixed reality technology, S1: first, algorithm initialization is performed, and initial distribution is obtained by using feature point matching and the first frame image, then the feature points with stable comparison are selected by the pose estimation value accuracy determination basis, and light flow tracking, three-dimensional reconstruction and ukf position estimation are sequentially performed;

[0024] S2: Then, whether to add to the three-dimensional point sequence of the pose estimation is determined by the position estimation criterion-projection error;

[0025] S3: Finally, the algorithm performs pose estimation based on the three-dimensional points. During the algorithm running process, when the scene points move out, the three-dimensional point sequence of the current pose estimation is deleted, and when there are new feature points, three-dimensional reconstruction is performed again. The stable new points are put into the three-dimensional point sequence for pose estimation after position estimation. Then, the center of gravity and search radius of the M feature point particle re-projection distribution are used to obtain the candidate measurement region corresponding to the M feature points.

[0026] S4: Steps S1-S3 are looped.

[0027] In the present embodiment, a simple and fast feature point extraction method-Fast is used to avoid time-consuming Gaussian difference filtering on the image to improve the real-time performance of the system.

[0028] In this embodiment, the system obtains a set of descriptors for the same feature point under different viewpoints and perspectives by performing scale and affine transformations on the reference image in an offline stage, so as to achieve the system's robustness to changes in user viewpoint and perspective.

[0029] In this embodiment, based on the feature point matching strategy of the simplified SIFT algorithm, the reference image is first scaled using Gaussian filtering, and then the appearance of the feature points under the change of viewing angle and viewpoint is simulated and a set of descriptors is generated using the affine transformation shown in Equation (1).

[0030] Equation (1)

[0031] Where x0 is the position of the feature point on the reference image, x′0 is the center of the transformed image, and x′ is the position of x after the transformation.

[0032] In this embodiment, in step S1, optical flow tracking, 3D reconstruction and UKF position estimation are performed sequentially. First, a reference frame and a reference system for feature points are selected, and then the position of the feature points in the corresponding reference frame is determined. The 3D point position of the corresponding image feature is estimated accordingly. Therefore, map construction can be divided into three main steps: selecting a reference frame, calculating the 3D position of the feature points in the reference frame, and map update control.

[0033] In this embodiment, in step S3, the measurement process includes the projection of the estimated mean k of the current estimated 3D scene points to the camera pose. The estimated value comes from particle filtering. First, the projection of feature point zm in the camera state is as shown in equation (2).

[0034] Equation (2)

[0035] Here, Π refers to pinhole imaging projection. Therefore, the measurement process only uses xk in the particle filter at time k, that is, the stability of the reprojection point is determined by the reprojection error constraint.

[0036] In this embodiment, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a main controller, implements the method as described in any one of the above descriptions.

[0037] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the invention by those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the invention should be included within the scope of protection of the invention.

Claims

1. A virtual-real synchronization method based on a mixed reality technology, characterized in that, The method is implemented according to the following steps: S1: Firstly, algorithm initialization is performed, and an initial distribution is obtained by using feature point matching and a first frame image, then stable feature points are selected according to a pose estimation value precision judgment basis, and optical flow tracking, three-dimensional reconstruction and ukf position estimation are sequentially performed; firstly, a reference frame and a reference system of feature points are selected, then the positions of the feature points in the corresponding reference frame are determined, and the three-dimensional point positions of the corresponding image features are estimated, based on which a feature point matching strategy of the SIFT algorithm is simplified, firstly, a reference image is subjected to a scale transformation by using a Gaussian filter, then an appearance of the feature points under a change of a view angle and a view point is simulated by using an affine transformation shown in formula (1), and a descriptor set is generated; Formula (1) Wherein, x0 is a position of the feature point on the reference image, x'0 is a transformed image center, and x' is a position of x after transformation; Therefore, map construction is divided into three main steps: selection of a reference frame, calculation of three-dimensional positions of feature points in the reference frame and map update control; S2: Then, whether to add to a three-dimensional point sequence of a pose estimation is judged by using a position estimation criterion-reprojection error; S3: Finally, the algorithm performs pose estimation according to three-dimensional points, in the process of algorithm running, when scene points move out, the three-dimensional point sequence of the current pose estimation is deleted, when new feature points are added, three-dimensional reconstruction is performed again, and the stable new points are put into the three-dimensional point sequence for pose estimation after position estimation; then, a gravity center of a particle projection distribution of M feature points and a search radius are used to obtain candidate measurement regions corresponding to the M feature points; the measurement process includes projection of a three-dimensional scene point of a current estimation to an estimated mean value k of a camera pose, and the estimated value is derived from a particle filter, and projection of a feature point zm in a camera state, and the projection is specifically shown in formula (2); Formula (2) Wherein, Π refers to pinhole imaging projection, therefore, in the process of performing measurement, only xk in the particle filter at the k moment is used, that is, the stability of the reprojected point is judged by using a reprojected error constraint; S4: Steps S1-S3 are cycled. 2.The virtual-real synchronous method based on mixed reality technology according to claim 1, characterized in that, A simple and fast feature point extraction method-Fast is used to avoid time-consuming Gaussian difference filtering on images to improve system real-time performance. 3.The virtual-real synchronous method based on mixed reality technology according to claim 1, characterized in that, By performing scale and affine transformation on the reference image in the offline stage, a descriptor set of the same feature point under different view points and view angles is obtained, so as to realize robustness of the system to changes of the user's view point and view angle.

4. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the host controller to implement the method according to any one of claims 1-3.

Images