A method for augmented reality three-dimensional registration and an electronic device

Abstract

The present application provides a method and an electronic device for augmented reality three-dimensional registration. The method includes the following steps: obtaining an initial position and an initial rotation angle of a virtual three-dimensional model; determining three planes of a physical object target and three planes of the virtual three-dimensional model according to user selection and a plane generation method, so as to determine a position offset and a rotation offset between the physical object target and the virtual three-dimensional model; determining a target position of the physical object target according to the initial position and the position offset; determining a target rotation angle of the physical object target according to the initial rotation angle and the rotation offset; updating the position of the virtual three-dimensional model to the target position and updating the rotation angle of the virtual three-dimensional model to the target rotation angle to complete the three-dimensional registration of augmented reality. The solution provided by the present application can complete the three-dimensional registration of augmented reality in cases of large-scale scenes, weak texture environments, and large target objects, and is applicable to large-scale application scenarios such as shipbuilding and construction engineering.

Description

Technical Field

[0001] The present invention relates to the technical field of augmented reality, and particularly to a method for three-dimensional registration of augmented reality and an electronic device. Background Art

[0002] Augmented Reality (AR) is a human-computer interaction technology that visually fuses virtual information with the real physical environment. This technology relies on computers and multi-sensors, virtual-real picture fusion, real-time interaction, and three-dimensional registration technology in virtual-real space to realize, and can update the position and posture of virtual information relative to the real physical world in real time, so as to present a credible interaction feedback and visual enhancement effect. Compared with Virtual Reality (VR), Augmented Reality focuses more on supplementing the information of the real physical environment rather than completely replacing the real physical environment, and is mainly used to enhance the user's perception of the real physical world and the ability to interact with it.

[0003] Three-dimensional registration is one of the important key technologies for realizing augmented reality. Its main task is to calculate the conversion relationship between the coordinate system of the camera and the coordinate system of the physical world according to the line-of-sight direction of the current camera, so as to correctly superimpose virtual information onto the real physical environment.

[0004] Traditional three-dimensional registration methods can be divided into two technical means based on computer vision and based on hardware sensors, and the two technical means will be used in combination in practical applications. However, these methods all have obvious limitations and are not applicable in large-scale application scenarios such as shipbuilding and construction projects. Summary of the Invention

[0005] The embodiments of the present application provide a method for three-dimensional registration of augmented reality and an electronic device, which can complete the three-dimensional registration of augmented reality in the case of large-scale scenes, weak texture environments, and large target objects, and are applicable to large-scale application scenarios such as shipbuilding and construction projects.

[0006] The embodiment of the present application provides a method for augmented reality three-dimensional registration, including the following steps: obtaining the initial position and initial rotation angle of a virtual three-dimensional model; determining three planes of a physical object target and three planes of the virtual three-dimensional model according to user selection and plane generation method; determining the position offset and rotation offset between the physical object target and the virtual three-dimensional model according to the three planes of the physical object target and the three planes of the virtual three-dimensional model; determining the target position of the physical object target according to the initial position and the position offset; determining the target rotation angle of the physical object target according to the initial rotation angle and the rotation offset; updating the position of the virtual three-dimensional model to the target position and updating the rotation angle of the virtual three-dimensional model to the target rotation angle to complete the three-dimensional registration of augmented reality.

[0007] In one embodiment, the step of determining three planes of a physical object target according to user selection and plane generation method includes: obtaining a first plane, a second plane and a third plane of the physical object target by using a plane generation method according to the physical world plane or physical world three-dimensional grid surface selected by the user; wherein, the first plane, the second plane and the third plane intersect pairwise and intersect at a point.

[0008] In one embodiment, the step of determining three planes of the virtual three-dimensional model according to user selection and plane generation method includes: obtaining a fourth plane, a fifth plane and a sixth plane of the virtual three-dimensional model by using a plane generation method according to the planes on the virtual three-dimensional model corresponding to the first plane, the second plane and the third plane; wherein, the fourth plane, the fifth plane and the sixth plane intersect pairwise and intersect at a point.

[0009] In one embodiment, the step of determining the position offset between the physical object target and the virtual three-dimensional model according to the three planes of the physical object target and the three planes of the virtual three-dimensional model includes: determining a first intersection line between the first plane and the second plane; determining a first intersection point between the third plane and the first intersection line; determining a second intersection line between the fourth plane and the fifth plane; determining a second intersection point between the sixth plane and the second intersection line; determining the position offset according to the first intersection point and the second intersection point.

[0010] In one embodiment, determining a rotational offset between the physical object and the virtual 3D model based on three planes of the physical object and three planes of the virtual 3D model includes: constructing a first rotation matrix according to a first normal vector of the first plane, a second normal vector of the second plane, and a third normal vector of the third plane; constructing a second rotation matrix according to a fourth normal vector of the fourth plane, a fifth normal vector of the fifth plane, and a sixth normal vector of the sixth plane; multiplying the second rotation matrix by the inverse matrix of the first rotation matrix to obtain a combined rotation matrix; converting the combined rotation matrix into a quaternion to obtain a rotational difference quaternion; and using the rotational difference quaternion as the rotational offset.

[0011] In one embodiment, determining a target position of the physical object according to the initial position and the position offset includes: adding the initial position and the position offset to obtain the target position.

[0012] In one embodiment, determining a target rotation angle of the physical object according to the initial rotation angle and the rotational offset includes: multiplying the initial rotation angle by the rotational offset to obtain the target rotation angle.

[0013] In one embodiment, the plane generation method includes: retrieving a plurality of second triangular mesh faces adjacent to a first triangular mesh face selected by a user; when all the triangular mesh faces approximate to be coplanar, determining the height of the center point of each second triangular mesh face from the first triangular mesh face and obtaining an average height; and creating a plane parallel to the first triangular mesh face at the average height in the normal direction of the center point of the first triangular mesh face.

[0014] In one embodiment, all the triangular mesh faces approximating to be coplanar includes: obtaining vertices of all the triangular mesh faces, where each triangular mesh face includes three vertices; determining a normalized normal vector of the first triangular mesh face according to the three vertices of the first triangular mesh face; selecting any one of the three vertices of the first triangular mesh face and forming a plurality of first vectors with the vertices of the plurality of second triangular mesh faces; determining whether all the triangular mesh faces approximate to be coplanar according to the normalized normal vector and the plurality of first vectors; and when a preset condition is satisfied between the normalized normal vector and the plurality of first vectors, determining that all the triangular mesh faces approximate to be coplanar.

[0015] An embodiment of the present application further provides an electronic device, which includes: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the above method for augmented reality 3D registration.

[0016] The solution provided in the above embodiments of the present application can effectively improve the problems of cumbersome operation and high cost in using augmented reality virtual-real registration technology in the industrial field, and can quickly register a virtual 3D model onto a physical object in a new site. It only needs to perform real-time scanning of Simultaneous Localization and Mapping (SLAM) on the environment, and then click a few times, without the need to manually adjust the position offsets and rotation offsets of the three axes tediously, nor rely on complex preliminary preparations such as physical recognition, contour recognition, and field point cloud scanning. Description of the Drawings

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required to be used in the embodiments of the present application will be briefly introduced below.

[0018] Figure 1 is a schematic structural diagram of an electronic device provided by an embodiment of the present application;

[0019] Figure 2 is a schematic flowchart of a method for augmented reality 3D registration provided by an embodiment of the present application. Detailed Embodiments

[0020] The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application.

[0021] Similar reference numerals and letters denote similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of the present application, terms such as "first", "second", etc. are only used for differential description and cannot be construed as indicating or implying relative importance.

[0022] As described above, the 3D registration method can be divided into two technical means based on computer vision and based on hardware sensors, and the two technical means will be used in combination in actual applications. Among them, the method based on computer vision can be further divided into two methods based on artificial markers (pictures) and markerless. The method based on hardware sensors uses various hardware sensors, such as cameras, lidar (LiDAR), depth sensors, inertial measurement units (IMU), etc., to improve the precise tracking and positioning of target objects in the real scene.

[0023] The method based on artificial markers has the advantages of low computational complexity, good real-time performance and accuracy. However, since it is necessary to pre-edit the position and attitude relationship of virtual information relative to the markers and arrange artificial markers at corresponding positions in the real physical environment, a large number of artificial markers are often required for a large-scale scene, resulting in a relatively complex and cumbersome overall operation process.

[0024] There are three technical routes for markerless-based methods. The first is a 3D registration method based on the natural visual features of the real physical environment. This method calculates the spatial pose of the camera through the matching relationship between feature point sets, and then completes 3D registration. However, this method requires frame-by-frame calculation, and there are problems such as high computational overhead, low registration frame rate, and poor pose calculation stability.

[0025] The second is a 3D registration method based on a 3D model. One path of this method is to calculate and obtain contour edge feature information by additionally introducing 3D model data, which can handle the situation where the target object lacks texture information or has no texture information. However, the camera must be far away from the target object to obtain contour edge feature information, which is not suitable for close-range observation and internal roaming of large target objects, such as building models. Another path is to represent the real physical world or the target object to be tracked by additionally introducing 3D model point cloud data, and calculate the spatial pose of the camera through the matching relationship between the environmental point cloud and the 3D model point cloud, and then complete 3D registration. The disadvantage is that the scanning and processing process of the environmental point cloud is relatively cumbersome and the cost is high.

[0026] The third is to perform 3D registration based on visual SLAM. This technology can start moving from an unknown position in an unknown environment, perform self-localization based on position estimation and map during the movement, and at the same time construct an incremental map based on self-localization, and realize inter-frame pose estimation and loop detection through the matching between feature points, so as to realize 3D registration. However, this method depends on the texture features of the environment, and the position and angle errors will continue to accumulate after continuous use, especially when moving over a large range, it is easy to cause the deviation of 3D registration.

[0027] In summary, in related technologies, the 3D registration method for augmented reality has obvious limitations in large-scale scenes, weak texture environments, and large target objects. For example, it is difficult to be applied to large-scale application scenarios such as shipbuilding and construction projects.

[0028] To solve the above problems, an embodiment of the present application provides a method for augmented reality three-dimensional registration, including the following steps: obtaining the initial position and initial rotation angle of a virtual three-dimensional model; determining three planes of a physical object target and three planes of the virtual three-dimensional model according to user selection and plane generation methods; determining the position offset and rotation offset between the physical object target and the virtual three-dimensional model according to the three planes of the physical object target and the three planes of the virtual three-dimensional model; determining the target position of the physical object target according to the initial position and the position offset; determining the target rotation angle of the physical object target according to the initial rotation angle and the rotation offset; updating the position of the virtual three-dimensional model to the target position and updating the rotation angle of the virtual three-dimensional model to the target rotation angle to complete the three-dimensional registration of augmented reality. Through the three-dimensional registration completed by this method, the cumbersome operation and high cost problems of using augmented reality virtual-real registration technology in the industrial field can be effectively improved. This method is convenient to use and can quickly register a virtual model to a physical object in a new site. Only need to perform real-time SLAM scanning on the environment, and then click a few times, without the need for cumbersome manual adjustment of the position offset and rotation offset of the three axes, nor relying on physical recognition, contour recognition, and field point cloud scanning that require complex preliminary preparations. At the same time, this method can correct the virtual-real registration offset problem caused by the SLAM cumulative error after large-scale movement in the industrial field at low cost and quickly, and quickly register the virtual model to the physical object. In the related art, augmented reality updates the cumulative changes of the three-dimensional coordinates and three-dimensional rotation of an augmented reality device in the physical world through a gyroscope, an optical lens, and even a lidar. Taking the iPad Pro as an example, for every one-meter movement of the user, an average positioning error of about 1 centimeter will be accumulated, and there is at least an angular error of 0.1 degrees. In industrial scenarios with products and sites of tens or hundreds of meters, it is very easy to generate virtual-real registration offsets. And this method can quickly recalibrate the virtual-real registration relationship by clicking on the plane to improve the positioning drift problem caused by the cumulative error.

[0029] Figure 1 It is a schematic structural diagram of an electronic device provided by an embodiment of the present application. The electronic device 100 can be an iOS device supporting ARKit or an Android device supporting ARcore, and is used to execute the method for augmented reality three-dimensional registration provided by an embodiment of the present application. As Figure 1 shown, the electronic device 100 includes: one or more processors 102, and one or more memories 104 for storing processor-executable instructions. Among them, the processor 102 is configured to execute the method for augmented reality three-dimensional registration provided in the following embodiments of the present application.

[0030] It should be noted especially that Figure 1The electronic device in [it] is only one possible device structure, and there are other possible device structures, which are not limited in this application.

[0031] The processor 102 can be a gateway, a smart terminal, or a device including a central processing unit (CPU), an image processing unit (GPU), or other forms of processing units with data processing capabilities and / or instruction execution capabilities. It can process the data of other components in the electronic device 100 and can also control other components in the electronic device 100 to perform desired functions.

[0032] The memory 104 can include one or more computer program products, and the computer program products can include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory can include, for example, random access memory (RAM) and / or cache memory, etc. The non-volatile memory can include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions can be stored on the computer-readable storage media, and the processor 102 can run the program instructions to implement the augmented reality three-dimensional registration method described below. Various application programs and various data can also be stored in the computer-readable storage media, such as various data used and / or generated by the application programs, etc.

[0033] In one embodiment, Figure 1 The illustrated electronic device 100 may further include an input device 106, an output device 108, and a data acquisition device 110. These components are interconnected through a bus system 112 and / or other forms of connection mechanisms (not shown). It should be noted that Figure 1 The components and structures of the illustrated electronic device 100 are exemplary, not restrictive. According to needs, the electronic device 100 may also have other components and structures.

[0034] The input device 106 can be a device for a user to input instructions and can include one or more of a keyboard, a mouse, a microphone, and a touch screen, etc. The output device 108 can output various information (such as images or sounds) to the outside (for example, to the user) and can include one or more of a display, a speaker, etc. The data acquisition device 110 can acquire a plane of the physical world or a three-dimensional grid surface of the physical world. Exemplarily, the data acquisition device 110 can be a camera.

[0035] In one embodiment, the devices in the exemplary electronic device 100 for implementing the method of augmented reality three-dimensional registration according to the embodiments of the present application may be integrally provided or separately provided. For example, the processor 102, the memory 104, the input device 106, and the output device 108 may be integrally provided, while the data acquisition device 110 is separately provided.

[0036] In one embodiment, the exemplary electronic device 100 for implementing the method of augmented reality three-dimensional registration according to the embodiments of the present application may be implemented as intelligent terminals such as AR hardware devices, VR hardware devices, MR (Mixed Reality) hardware devices, XR (Extended Reality), smart phones, tablets, desktop computers, servers, in-vehicle devices, etc.

[0037] Figure 2 It is a schematic flowchart of a method for augmented reality three-dimensional registration according to an embodiment of the present application. As Figure 2 shown, the method includes the following steps 210 - step 260.

[0038] Step 210: Obtain the initial position and the initial rotation angle of the virtual three-dimensional model.

[0039] The embodiments of the present application rely on iOS devices supporting ARKit or Android devices supporting ARcore because these devices can detect the planes of the physical world or establish a three-dimensional grid surface of the physical world.

[0040] When the user activates the augmented reality function, the position and rotation of the device in the physical world coordinate system are both 0. Through the augmented reality function, the device updates its latest position and rotation values in the physical world coordinate system in real time.

[0041] Denote the initial position of the virtual three-dimensional model displayed in the physical world coordinate system as OriginPosition, the initial rotation angle as OriginRotation, the target position of the physical object target in the physical space as TargetPosition, and the target rotation angle as TargetRotation. The goal of virtual and real three-dimensional registration is to calculate the difference between Origin and Target and superimpose the virtual three-dimensional model on the physical object target.

[0042] For the initial position and the initial rotation angle of the virtual three-dimensional model, they can be obtained through the augmented reality function. For the target position and the target rotation angle of the physical object target, they can be obtained through the following steps 220 - 250.

[0043] Step 220: Determine three planes of the physical object target and three planes of the virtual three-dimensional model according to the user selection and the plane generation method.

[0044] On the one hand, determining three planes of a physical object according to user selection and a plane generation method includes: obtaining a first plane, a second plane, and a third plane of the physical object by using the plane generation method based on the physical world plane or the physical world three-dimensional grid surface selected by the user; wherein, the first plane, the second plane, and the third plane intersect pairwise and intersect at a point.

[0045] Specifically, the three planes of the physical object can be determined through the following steps 2201-2205.

[0046] Step 2201: The user selects the physical world plane or the physical world three-dimensional grid surface recognized by augmented reality, and the first plane: Plane A is obtained through calculation by the plane generation method. The plane generation method can be a plane generation algorithm, which will be introduced in detail later and will not be elaborated here.

[0047] Step 2202: The user continues to select the physical world plane or the physical world three-dimensional grid surface recognized by augmented reality, and the second plane: Plane B is obtained through calculation by the plane generation method.

[0048] Step 2203: Verify whether Plane A and Plane B intersect. If the condition is not met, return to Step 2202.

[0049] Step 2204: The user continues to select the physical world plane or the physical world three-dimensional grid surface recognized by augmented reality, and the third plane: Plane C is obtained through calculation by the plane generation method.

[0050] Step 2205: Verify whether the three planes A, B, and C intersect pairwise and intersect at a point. If the condition is not met, return to Step 2204.

[0051] On the other hand, determining three planes of a virtual three-dimensional model according to user selection and a plane generation method includes: obtaining a fourth plane, a fifth plane, and a sixth plane of the virtual three-dimensional model by using the plane generation method based on the plane on the virtual three-dimensional model corresponding to the first plane, the second plane, and the third plane selected by the user; wherein, the fourth plane, the fifth plane, and the sixth plane intersect pairwise and intersect at a point.

[0052] Specifically, the three planes of the virtual three-dimensional model can be determined through the following steps 2206-2210.

[0053] Step 2206: The user selects the plane A' on the virtual three-dimensional model placed in the physical world and corresponding to Plane A, and the fourth plane: Plane A' is obtained through calculation by the plane generation method.

[0054] Step 2207: The user continues to select the plane B' corresponding to the B surface on the virtual three-dimensional model placed in the physical world, and the fifth plane is calculated through the plane generation method: the B' plane.

[0055] Step 2208: Verify whether the A' and B' planes intersect. If the condition is not met, go to Step 2207.

[0056] Step 2209: The user continues to select the plane C' corresponding to the C surface on the virtual three-dimensional model placed in the physical world, and the sixth plane is calculated through the plane generation method: the C' plane.

[0057] Step 2210: Verify whether the three planes A', B', and C' intersect pairwise and intersect at a point. If the condition is not met, go to Step 2210.

[0058] Step 230: Determine the position offset and rotation offset between the physical object target and the virtual three-dimensional model according to the three planes of the physical object target and the three planes of the virtual three-dimensional model.

[0059] On the one hand, the determining the position offset between the physical object target and the virtual three-dimensional model according to the three planes of the physical object target and the three planes of the virtual three-dimensional model includes: determining the first intersection line between the first plane and the second plane; determining the first intersection point between the third plane and the first intersection line; determining the second intersection line between the fourth plane and the fifth plane; determining the second intersection point between the sixth plane and the second intersection line; determining the position offset according to the first intersection point and the second intersection point. Further, the value obtained by subtracting the second intersection point from the first intersection point is used as the position offset.

[0060] Specifically, the position offset between the physical object target and the virtual three-dimensional model can be determined through the following steps 2301 - 2305.

[0061] Step 2301: Obtain the intersection line line of the AB plane according to the A and B surfaces.

[0062] Step 2302: Use line to obtain the intersection point p with the C surface.

[0063] Step 2303: Obtain the intersection line line' of the A'B' plane according to the A' and B' surfaces.

[0064] Step 2304: Use line' to obtain the intersection point p' with the C' surface.

[0065] Step 2305: p - p' is the position offset vector.

[0066] On the other hand, determining the rotational offset between the physical target and the virtual three-dimensional model based on the three planes of the physical target and the three planes of the virtual three-dimensional model includes: constructing a first rotation matrix according to the first normal of the first plane, the second normal of the second plane, and the third normal of the third plane; constructing a second rotation matrix according to the fourth normal of the fourth plane, the fifth normal of the fifth plane, and the sixth normal of the sixth plane; multiplying the second rotation matrix by the inverse matrix of the first rotation matrix to obtain a combined rotation matrix; converting the combined rotation matrix into a quaternion to obtain a rotation difference quaternion; and using the rotation difference quaternion as the rotational offset.

[0067] Specifically, the rotational offset between the physical target and the virtual three-dimensional model can be determined through the following steps 2306-2309.

[0068] Step 2306: Construct a rotation matrix M according to the normal vectors of the three planes A, B, and C (which are v1, v2, and v3 in sequence), specifically as follows:

[0069]

[0070] Step 2307: Construct a rotation matrix M' according to the normal vectors of the three planes A', B', and C' (which are v'1, v'2, and v'3 in sequence), specifically as follows:

[0071]

[0072] Step 2308: Multiply M' by the inverse matrix of M to calculate the combined rotation matrix, and then convert it into a quaternion to obtain the rotation difference quaternion R, specifically as follows:

[0073] R = Quaternion(M′×M -1 )

[0074] Step 2309: Use R as the rotational offset.

[0075] Step 240: Determine the target position of the physical target according to the initial position and the position offset.

[0076] Specifically, add the initial position and the position offset to obtain the target position, as follows:

[0077] TargetPos i t ion = OriginPos i t ion + p - p’

[0078] Step 250: Determine the target rotation angle of the physical target according to the initial rotation angle and the rotational offset.

[0079] Specifically, multiply the initial rotation angle by the rotation offset to obtain the target rotation angle as follows:

[0080] TargetRotation = R × OriginRotation

[0081] Step 260: Update the position of the virtual 3D model to the target position and update the rotation angle of the virtual 3D model to the target rotation angle to complete the 3D registration of augmented reality.

[0082] The plane generation method involved in the above steps is the key point for generating the plane required for 3D registration in the embodiments of the present application and can be implemented using a plane generation algorithm, including: retrieving a plurality of second triangular mesh grids adjacent to the first triangular mesh grid according to the first triangular mesh grid selected by the user; when all the triangular mesh grids approach coplanarity, determining the height of the center point of each second triangular mesh grid from the first triangular mesh grid and obtaining the average height; creating a plane parallel to the first triangular mesh grid at the average height in the normal direction of the center point of the first triangular mesh grid. Among them, the fact that all the triangular mesh grids approach coplanarity includes: obtaining the vertices of all the triangular mesh grids, where each triangular mesh grid includes three vertices; determining the normalized normal vector of the first triangular mesh grid according to the three vertices of the first triangular mesh grid; selecting any one of the three vertices of the first triangular mesh grid and forming a plurality of first vectors with the vertices of the plurality of second triangular mesh grids; judging whether all the triangular mesh grids approach coplanarity according to the normalized normal vector and the plurality of first vectors; when a preset condition is satisfied between the normalized normal vector and the plurality of first vectors, determining that all the triangular mesh grids approach coplanarity.

[0083] Regarding judging whether all the triangular mesh grids approach coplanarity according to the normalized normal vector and the plurality of first vectors; when a preset condition is satisfied between the normalized normal vector and the plurality of first vectors, determining that all the triangular mesh grids approach coplanarity, specifically including: normalizing the plurality of first vectors to obtain a plurality of normalized first vectors; multiplying the plurality of normalized first vectors by the normalized normal vector in sequence and taking the absolute value. If these values all satisfy the preset condition, then it can be determined that all the triangular mesh grids approach coplanarity.

[0084] Specifically, when the user selects a virtual 3D model in the augmented reality screen, a physical world plane detected and recognized by the augmented reality plane detection, or a 3D mesh for modeling the physical world by the augmented reality, a triangular mesh grid Tri1 will be selected. According to Tri1, a plurality of adjacent triangular mesh grids Tri 2～n , and judge Tri 1～nWhether it is close to coplanarity. If it is close to coplanarity, calculate Tri 2～n The arithmetic mean h of the distances from the center points to Tri1 (i.e., the average height. First, calculate the height of each center point from Tri1, and then calculate the arithmetic mean of all the heights to obtain h). Create a plane parallel to Tri1 at a height of h in the normal direction of the center point of Tri1, which is the required plane.

[0085] The above judgment of Tri 1～n The method for determining whether it is close to coplanarity is as follows:

[0086] Define all the vertices on the triangular mesh Tri 1～n as p1, p2, p3... p n , and the error coefficient k (empirical parameter) is cos85°, that is, an error of 5°.

[0087] If n = 3, it is directly considered coplanar. If n ≥ 4, enter the subsequent calculation.

[0088] The three points p1, p2, and p3 form a plane (i.e., the first triangular mesh Tri1 in the above embodiment), and the normalized vector of its normal vector (i.e., the normalized normal vector of Tri1) is specifically:

[0089]

[0090] p4, p5... p n are the vertices of multiple second triangular meshes Tri 2～n in the above embodiment, that is, starting from p4 to p n Calculate in sequence to determine whether Tri 1～n is close to coplanarity.

[0091] If all then it is considered that Tri 1～n is close to coplanarity, otherwise it is considered that Tri 1～n is not close to coplanarity.

[0092] The above embodiment can effectively improve the problems of cumbersome operation and high cost in using augmented reality virtual-real registration technology in the industrial field, and can quickly register a virtual model to a physical object in a new site. Only need to perform real-time SLAM scanning on the environment, and then click a few times, without the need for cumbersome manual adjustment of the position offset and rotation offset of the three axes, nor relying on complex preparatory work such as physical object recognition, contour recognition, and field point cloud scanning.

[0093] In several embodiments provided in this application, the disclosed devices and methods can also be implemented in other ways. The device and method embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions, and operations of devices, methods, and computer program products according to multiple embodiments of this application. In this regard, each block in the flowchart or block diagram can represent a module, a program segment, or a part of code, and a module, a program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than that marked in the accompanying drawings. For example, two consecutive blocks can actually be executed substantially in parallel, and they can sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, as well as the combination of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0094] In addition, each functional module in various embodiments of this application can be integrated together to form an independent part, or each module can exist alone, or two or more modules can be integrated to form an independent part.

[0095] If the function is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in various embodiments of this application. The aforementioned storage medium includes: USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs, etc., which can store program codes.

Claims

1. A method for augmented reality three-dimensional registration, characterized in that: The following steps are involved: Obtaining the initial position and initial rotation angle of the virtual three-dimensional model; Determine three planes of the physical object and three planes of the virtual three-dimensional model according to user selection and plane generation method; Determining a position offset and a rotation offset between the physical object and the virtual three-dimensional model according to three planes of the physical object and three planes of the virtual three-dimensional model; Determining the target position of the physical object according to the initial position and the position offset; Determining a target rotation angle of the physical object according to the initial rotation angle and the rotation offset; Updating the position of the virtual three-dimensional model to the target position, and updating the rotation angle of the virtual three-dimensional model to the target rotation angle, so as to complete the three-dimensional registration of augmented reality; The three planes of the physical object are determined according to the user selection and the plane generation method, including: According to the physical world plane or the physical world three-dimensional grid surface selected by the user, a first plane, a second plane and a third plane of the physical object are obtained by using a plane generation method; Wherein, the first plane, the second plane and the third plane intersect each other and intersect at one point; Determining three planes of the virtual three-dimensional model according to the user selection and the plane generation method includes: According to the planes on the virtual three-dimensional model corresponding to the first plane, the second plane and the third plane selected by the user, a fourth plane, a fifth plane and a sixth plane of the virtual three-dimensional model are obtained by using a plane generation method; Wherein, the fourth plane, the fifth plane and the sixth plane intersect each other and intersect at one point; The determining of the position offset between the physical object and the virtual three-dimensional model according to the three planes of the physical object and the three planes of the virtual three-dimensional model comprises: determining a first intersection line between the first plane and the second plane; determining a first intersection point between the third plane and the first intersection line; determining a second intersection line between the fourth plane and the fifth plane; determining a second intersection point between the sixth plane and the second intersection line; Determine the position offset according to the first intersection point and the second intersection point; Determining the rotation offset between the physical object and the virtual three-dimensional model according to the three planes of the physical object and the three planes of the virtual three-dimensional model includes: constructing a first rotation matrix according to a first normal of the first plane, a second normal of the second plane, and a third normal of the third plane; constructing a second rotation matrix according to the fourth normal of the fourth plane, the fifth normal of the fifth plane, and the sixth normal of the sixth plane; Multiply the second rotation matrix by the inverse matrix of the first rotation matrix to obtain a combined rotation matrix; Convert the combined rotation matrix into a quaternion to obtain a rotation difference quaternion; The rotation difference quaternion is used as the rotation offset.

2. The method for augmented reality three-dimensional registration according to claim 1, characterized in that: Determining the target position of the physical object according to the initial position and the position offset includes: The initial position is added to the position offset to obtain the target position.

3. The method for augmented reality three-dimensional registration according to claim 1, characterized in that: Determining a target rotation angle of the physical object according to the initial rotation angle and the rotation offset includes: The target rotation angle is obtained by multiplying the initial rotation angle by the rotation offset.

4. The method for augmented reality three-dimensional registration according to claim 1, characterized in that: The plane generation method includes: According to the first triangular surface mesh selected by the user, a plurality of second triangular surface meshes adjacent to the first triangular surface mesh are retrieved; When all triangular surface meshes are close to being coplanar, determine the height of the center point of each second triangular surface mesh from the first triangular surface mesh, and obtain an average height; A plane parallel to the first triangular surface mesh is created in the normal direction of the center point of the first triangular surface mesh and at the average height.

5. The method for augmented reality three-dimensional registration according to claim 4, characterized in that: All triangular meshes are approximately coplanar, including: Obtaining vertices of all the triangular meshes, wherein each triangular mesh includes three vertices; Determining a normalized normal vector of the first triangular surface mesh according to three vertices of the first triangular surface mesh; Select any one of the three vertices of the first triangular surface mesh to form a plurality of first vectors with the vertices of the plurality of second triangular surface meshes; Determining whether all the triangular surface meshes are approximately coplanar according to the normalized normal vector and the plurality of first vectors; When the normalized normal vector and a plurality of first vectors satisfy a preset condition, it is determined that all the triangular surface meshes are approximately coplanar.

6. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to execute the method for augmented reality three-dimensional registration as described in any one of claims 1-5.

Images