Augmented reality three-dimensional registration method and electronic equipment
By determining and updating the position and rotation angle of the virtual three-dimensional model, the problems of cumbersome and high cost in large-scale application scenarios are solved, and fast and low-cost virtual model registration in the fields of ship construction and construction engineering are achieved.
Patent Information
- Application Number
- CN202411235422.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-04
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2044-09-04
AI Technical Summary
The existing augmented reality three-dimensional registration technology has limitations in large-scale scenarios, weak texture environments and large-scale target objects, and is difficult to be applied to large-scale application scenarios such as ship construction and construction projects. It is cumbersome and costly.
By obtaining the initial position and rotation angle of the virtual 3D model, determine the position and rotation offset between the real object and the virtual 3D model according to the user selection and plane generation method, update the position and rotation angle of the virtual 3D model to complete augmented reality 3D registration simplifies operational processes and reduces costs.
It realizes the rapid and low-cost registration of virtual three-dimensional models to the real object in large-scale application scenarios, reduces the need for manual adjustment, corrects SLAM cumulative errors, and improves the accuracy and efficiency of virtual and real registration.
Smart Images

Figure CN119048718A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of augmented reality, and in particular to a method and electronic equipment for augmented reality three-dimensional registration. Background Art
[0002] Augmented Reality (AR) is a human-computer interaction technology that visually integrates virtual information with the real physical environment. This technology relies on computers and multiple sensors, virtual-real image fusion, real-time interaction, and virtual-real space three-dimensional registration technology. It can update the position and posture of virtual information relative to the real physical world in real time, thereby presenting credible interactive feedback and visual enhancement effects. Compared with Virtual Reality (VR), AR focuses more on supplementing the information of the real physical environment rather than completely replacing it. It is mainly used to enhance users' perception of the real physical world and their ability to interact with it.
[0003] Three-dimensional registration is one of the important key technologies for realizing augmented reality. Its main task is to calculate the transformation relationship between the camera's coordinate system and the physical world coordinate system according to the current camera's line of sight, so as to correctly superimpose the virtual information into the real physical environment.
[0004] Traditional 3D registration methods can be divided into two technical means: computer vision-based and hardware sensor-based. The two technical means are often combined in practical applications. However, these methods have obvious limitations and are not applicable in large-scale application scenarios such as shipbuilding and construction engineering. Summary of the invention
[0005] The embodiments of the present application provide a method and electronic device for augmented reality three-dimensional registration, which can complete augmented reality three-dimensional registration in large-scale scenes, weak texture environments, and large target objects, and is suitable for large-scale application scenarios such as shipbuilding and construction projects.
[0006] An embodiment of the present application provides a method for augmented reality three-dimensional registration, comprising the following steps: obtaining an initial position and an initial rotation angle of a virtual three-dimensional model; determining three planes of a physical target and three planes of a virtual three-dimensional model according to a user selection and a plane generation method; determining a position offset and a rotation offset between the physical target and the virtual three-dimensional model according to the three planes of the physical target and the three planes of the virtual three-dimensional model; determining a target position of the physical target according to the initial position and the position offset; determining a target rotation angle of the physical target according to the initial rotation angle and the rotation offset; updating the position of the virtual three-dimensional model to the target position, and updating the rotation angle of the virtual three-dimensional model to the target rotation angle, so as to complete augmented reality three-dimensional registration.
[0007] In one embodiment, determining three planes of the physical target based on user selection and a plane generation method includes: obtaining a first plane, a second plane, and a third plane of the physical target using a plane generation method based on a physical world plane or a physical world three-dimensional grid surface selected by the user; wherein the first plane, the second plane, and the third plane intersect with each other and intersect at one point.
[0008] In one embodiment, determining three planes of the virtual three-dimensional model based on user selection and a plane generation method includes: obtaining a fourth plane, a fifth plane, and a sixth plane of the virtual three-dimensional model using a plane generation method based on planes on the virtual three-dimensional model corresponding to the first plane, the second plane, and the third plane selected by the user; wherein the fourth plane, the fifth plane, and the sixth plane intersect with each other and intersect at one point.
[0009] In one embodiment, determining the position offset between the physical object and the virtual three-dimensional model based on the three planes of the physical object and the three planes of the virtual three-dimensional model includes: determining a first intersection line between the first plane and the second plane; determining a first intersection point between the third plane and the first intersection line; determining a second intersection line between the fourth plane and the fifth plane; determining a second intersection point between the sixth plane and the second intersection line; and determining the position offset based on the first intersection point and the second intersection point.
[0010] In one embodiment, determining the rotation offset between the physical target and the virtual three-dimensional model based on the three planes of the physical target and the three planes of the virtual three-dimensional model includes: constructing a first rotation matrix based on the first normal of the first plane, the second normal of the second plane, and the third normal of the third plane; constructing a second rotation matrix based on the fourth normal of the fourth plane, the fifth normal of the fifth plane, and the sixth normal of the sixth plane; multiplying the second rotation matrix by the inverse matrix of the first rotation matrix to obtain a combined rotation matrix; converting the combined rotation matrix into a quaternion to obtain a rotation difference quaternion; and using the rotation difference quaternion as the rotation offset.
[0011] In one embodiment, determining the target position of the physical object according to the initial position and the position offset includes: adding the initial position and the position offset to obtain the target position.
[0012] In one embodiment, determining the target rotation angle of the physical object according to the initial rotation angle and the rotation offset includes: multiplying the initial rotation angle by the rotation offset to obtain the target rotation angle.
[0013] In one embodiment, the plane generation method includes: based on a first triangular mesh selected by a user, retrieving multiple second triangular meshes adjacent to the first triangular mesh; when all triangular meshes are approximately coplanar, determining the height of the center point of each second triangular mesh from the first triangular mesh and obtaining the average height; and creating a plane parallel to the first triangular mesh in the normal direction of the center point of the first triangular mesh and at the average height.
[0014] In one embodiment, all the triangular meshes are close to being coplanar, including: obtaining the vertices of all the triangular meshes, wherein each triangular mesh includes three vertices; determining the normalized normal vector of the first triangular mesh based on the three vertices of the first triangular mesh; selecting any one of the three vertices of the first triangular mesh to form a plurality of first vectors with the vertices of the plurality of second triangular meshes; judging whether all the triangular meshes are close to being coplanar based on the normalized normal vector and the plurality of first vectors; and determining that all the triangular meshes are close to being coplanar when a preset condition is satisfied between the normalized normal vector and the plurality of first vectors.
[0015] An embodiment of the present application also provides an electronic device, comprising: a processor; a memory for storing processor executable instructions; wherein the processor is configured to execute the above-mentioned augmented reality three-dimensional registration method.
[0016] The solution provided by the above embodiment of the present application can effectively improve the cumbersome operation and high cost of using augmented reality virtual-real registration technology in the industrial field, and can quickly register the virtual three-dimensional model to the real object in a new site. It only needs to perform real-time positioning and mapping (SLAM) real-time scanning of the environment, and then a few clicks are enough, without the need to manually adjust the position offset and rotation offset of the three axes, and without relying on complex preliminary preparations such as real object recognition, contour recognition, and site cloud scanning. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] In order to more clearly illustrate the technical solution of the embodiments of the present application, the drawings required for use in the embodiments of the present application are briefly introduced below.
[0018] Figure 1 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;
[0019] Figure 2 It is a flowchart of a method for augmented reality three-dimensional registration provided in an embodiment of the present application. DETAILED DESCRIPTION
[0020] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.
[0021] Similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.
[0022] As mentioned above, 3D registration methods can be divided into two technical means: computer vision-based and hardware sensor-based. The two technical means will be used in combination in practical applications. Among them, the computer vision-based method can be subdivided into two methods: artificial markers (pictures)-based and marker-free. The hardware sensor-based method uses various hardware sensors, such as cameras, laser radar (LiDAR), depth sensors, inertial measurement units (IMU), etc., to improve the precise tracking and positioning of target objects in real scenes.
[0023] The method based on artificial markers has the advantages of low computational complexity, good real-time performance and accuracy. However, it requires pre-editing of the position and posture relationship of virtual information relative to the marker, and the arrangement of artificial markers at corresponding positions in the real physical environment. Large-scale scenes often require the arrangement of a large number of artificial markers, which makes the overall operation process more complicated and cumbersome.
[0024] There are three technical routes based on the marker-free method. The first is a 3D registration method based on the natural visual features of the real physical environment. This method calculates the spatial pose of the camera through the matching relationship between the feature point sets, and then completes the 3D registration. However, this method requires frame-by-frame calculation, and has problems such as high computational overhead, low registration frame rate, and poor pose calculation stability.
[0025] The second is a 3D registration method based on a 3D model. One path of this method is to calculate and obtain the contour edge feature information by additionally introducing 3D model data. This method can cope with the situation where the target object lacks texture information or has no texture information. However, the camera must be far away from the target object to obtain the contour edge feature information. It is not suitable for close observation and internal roaming of large target objects, such as building models. Another path is to introduce additional 3D model point cloud data to represent the real physical world or the target object to be tracked, calculate the spatial position of the camera through the matching relationship between the environmental point cloud and the 3D model point cloud, and then complete the 3D registration. The disadvantage is that the scanning and processing process of the environmental point cloud is relatively cumbersome and costly.
[0026] The third is 3D registration based on visual SLAM. This technology can start moving from an unknown position in an unknown environment, and locate itself according to the position estimation and map during the movement. At the same time, it builds an incremental map based on its own positioning, and realizes inter-frame pose estimation and closed-loop detection through matching between feature points, thereby realizing 3D registration. However, this method relies on the texture features of the environment, and will continue to accumulate position and angle errors after continuous use, especially after moving over a large range, which is easy to cause 3D registration offset.
[0027] To sum up, in the related technologies, the three-dimensional registration method of augmented reality has obvious limitations in large-scale scenes, weak-texture environments, and large target objects. For example, it is difficult to apply to large-scale application scenarios such as shipbuilding and construction projects.
[0028] In order to solve the above problems, the embodiment of the present application provides a method for augmented reality three-dimensional registration, comprising the following steps: obtaining the initial position and initial rotation angle of the virtual three-dimensional model; determining the three planes of the physical target and the three planes of the virtual three-dimensional model according to the user selection and the plane generation method; determining the position offset and rotation offset between the physical target and the virtual three-dimensional model according to the three planes of the physical target and the three planes of the virtual three-dimensional model; determining the target position of the physical target according to the initial position and the position offset; determining the target rotation angle of the physical target according to the initial rotation angle and the rotation offset; updating the position of the virtual three-dimensional model to the target position, and updating the rotation angle of the virtual three-dimensional model to the target rotation angle, so as to complete the augmented reality three-dimensional registration. The three-dimensional registration completed by this method can effectively improve the cumbersome operation and high cost problems of using augmented reality virtual-real registration technology in the industrial field. This method is easy to use and can quickly register the virtual model to the physical object in a new venue. It is only necessary to perform a SLAM real-time scan of the environment, and then click a few times. There is no need to manually adjust the position offset and rotation offset of the three axes, nor is there any need to rely on physical object recognition, contour recognition, and site cloud scanning, which require complex preliminary preparations. At the same time, this method can quickly and cheaply correct the virtual-real registration offset problem caused by the SLAM cumulative error after large-scale movement in the industrial field, and quickly register the virtual model to the physical object. In related technologies, augmented reality uses gyroscopes, optical lenses, and even laser radars to update the three-dimensional coordinates and three-dimensional rotation of augmented reality devices in the physical world in real time. Taking the iPad Pro as an example, for every meter the user moves, an average positioning error of about 1 cm will be accumulated, and there will be at least an angle error of 0.1 degrees. It is very easy to produce virtual-real registration offsets for products and sites that are ten meters or hundreds of meters in industrial scenes. This method can quickly recalibrate the virtual-real registration relationship by clicking on the plane, and improve the positioning drift problem caused by cumulative errors.
[0029] Figure 1 1 is a schematic diagram of the structure of the electronic device provided in the embodiment of the present application. The electronic device 100 may be an iOS device supporting ARKit or an Android device supporting ARcore, and is used to execute the method for augmented reality three-dimensional registration provided in the embodiment of the present application. Figure 1 As shown, the electronic device 100 includes: one or more processors 102, and one or more memories 104 storing processor executable instructions. The processor 102 is configured to execute the method for augmented reality three-dimensional registration provided in the following embodiment of the present application.
[0030] It is important to note that Figure 1The electronic device in is only one possible device structure, and there are other possible device structures, which are not limited in this application.
[0031] The processor 102 can be a gateway, a smart terminal, or a device including a central processing unit (CPU), a graphics processing unit (GPU), or other forms of processing units with data processing capabilities and / or instruction execution capabilities. It can process data of other components in the electronic device 100 and control other components in the electronic device 100 to perform desired functions.
[0032] The memory 104 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 102 may run the program instructions to implement the method of augmented reality three-dimensional registration described below. Various applications and various data, such as various data used and / or generated by the application, may also be stored in the computer-readable storage medium.
[0033] In one embodiment, Figure 1 The electronic device 100 shown may also include an input device 106, an output device 108, and a data acquisition device 110, which are interconnected via a bus system 112 and / or other forms of connection mechanisms (not shown). Figure 1 The components and structures of the electronic device 100 shown are merely exemplary and non-limiting. The electronic device 100 may also have other components and structures as required.
[0034] The input device 106 may be a device used by a user to input instructions, and may include one or more of a keyboard, a mouse, a microphone, a touch screen, etc. The output device 108 may output various information (e.g., images or sounds) to the outside (e.g., a user), and may include one or more of a display, a speaker, etc. The data acquisition device 110 may acquire a plane of the physical world or a three-dimensional grid surface of the physical world. Exemplarily, the data acquisition device 110 may be a camera.
[0035] In one embodiment, the various components in the example electronic device 100 for implementing the method of augmented reality three-dimensional registration of the embodiment of the present application can be integrated or dispersed, such as integrating the processor 102, memory 104, input device 106 and output device 108 into one, while separating the data acquisition device 110.
[0036] In one embodiment, the example electronic device 100 for implementing the method of augmented reality three-dimensional registration of the embodiment of the present application can be implemented as an intelligent terminal such as an AR hardware device, a VR hardware device, an MR (Mixed Reality) hardware device, an XR (Extended Reality), a smart phone, a tablet computer, a desktop computer, a server, a vehicle-mounted device, etc.
[0037] Figure 2 FIG. 1 is a flow chart of a method for augmented reality three-dimensional registration according to an embodiment of the present application. Figure 2 As shown, the method includes the following steps 210-260.
[0038] Step 210: Obtain the initial position and initial rotation angle of the virtual three-dimensional model.
[0039] The embodiments of the present application rely on iOS devices that support ARKit or Android devices that support ARcore, because these devices can detect planes in the physical world or build three-dimensional mesh surfaces in the physical world.
[0040] When the user starts the augmented reality function, the position and rotation of the device in the physical world coordinate system are both 0. Through the augmented reality function, the device updates its latest position and rotation values in the physical world coordinate system in real time.
[0041] The initial position of the virtual 3D model displayed in the physical world coordinate system is recorded as OriginPosition, the initial rotation angle is recorded as OriginRotation, the target position of the physical target in the physical space is recorded as TargetPosition, and the target rotation angle is recorded as TargetRotation. The goal of virtual-real 3D registration is to calculate the difference between Origin and Target, and superimpose the virtual 3D model on the physical target.
[0042] The initial position and initial rotation angle of the virtual three-dimensional model can be obtained through the augmented reality function, and the target position and target rotation angle of the physical object can be obtained through the following steps 220-250.
[0043] Step 220: Determine three planes of the physical object and three planes of the virtual three-dimensional model according to the user selection and the plane generation method.
[0044] On the one hand, the three planes of the physical target are determined according to the user selection and the plane generation method, including: according to the physical world plane or the physical world three-dimensional grid surface selected by the user, a plane generation method is used to obtain the first plane, the second plane and the third plane of the physical target; wherein the first plane, the second plane and the third plane intersect with each other and intersect at one point.
[0045] Specifically, the three planes of the physical target can be determined through the following steps 2201-2205.
[0046] Step 2201: The user selects a physical world plane or a physical world three-dimensional grid surface recognized by augmented reality, and calculates a first plane: plane A by plane generation method. The plane generation method may be a plane generation algorithm, which will be described in detail later and will not be repeated here.
[0047] Step 2202: The user continues to select a physical world plane or a physical world three-dimensional grid surface recognized by augmented reality, and calculates a second plane: plane B through a plane generation method.
[0048] Step 2203: Verify whether faces A and B intersect. If not, return to step 2202.
[0049] Step 2204: The user continues to select the physical world plane or the physical world three-dimensional grid surface recognized by the augmented reality, and calculates the third plane: C plane through the plane generation method.
[0050] Step 2205: Verify whether the three planes A, B, and C intersect each other and at one point. If the condition is not met, return to step 2204.
[0051] On the other hand, the method of determining the three planes of the virtual three-dimensional model based on the user selection and the plane generation method includes: obtaining the fourth plane, the fifth plane and the sixth plane of the virtual three-dimensional model by the plane generation method according to the planes on the virtual three-dimensional model corresponding to the first plane, the second plane and the third plane selected by the user; wherein the fourth plane, the fifth plane and the sixth plane intersect with each other and intersect at one point.
[0052] Specifically, the three planes of the virtual three-dimensional model may be determined through the following steps 2206-2210.
[0053] Step 2206: the user selects a plane A' that is placed on the virtual three-dimensional model in the physical world and corresponds to the A surface, and calculates a fourth plane: the A' surface through a plane generation method.
[0054] Step 2207: the user continues to select the plane B' that has been placed on the virtual three-dimensional model in the physical world and corresponds to the B surface, and calculates the fifth plane: the B' surface through the plane generation method.
[0055] Step 2208: Verify whether the A' and B' surfaces intersect. If not, proceed to step 2207.
[0056] Step 2209: the user continues to select the plane C' that has been placed on the virtual three-dimensional model in the physical world and corresponds to the C surface, and calculates the sixth plane: the C' surface through the plane generation method.
[0057] Step 2210: Verify whether the three planes A', B', and C' intersect each other and intersect at one point. If the condition is not met, then proceed to step 2210.
[0058] Step 230: Determine the position offset and rotation offset between the physical object and the virtual three-dimensional model according to the three planes of the physical object and the three planes of the virtual three-dimensional model.
[0059] On the one hand, the determining of the position offset between the physical object and the virtual three-dimensional model based on the three planes of the physical object and the three planes of the virtual three-dimensional model includes: determining a first intersection line between the first plane and the second plane; determining a first intersection point between the third plane and the first intersection line; determining a second intersection line between the fourth plane and the fifth plane; determining a second intersection point between the sixth plane and the second intersection line; and determining the position offset based on the first intersection point and the second intersection point. Specifically, the value of the first intersection point minus the second intersection point is taken as the position offset.
[0060] Specifically, the position offset between the physical object and the virtual three-dimensional model may be determined through the following steps 2301-2305.
[0061] Step 2301: Find the intersection line of plane AB based on planes A and B.
[0062] Step 2302: Use line to find the intersection point p with surface C.
[0063] Step 2303: Calculate the intersection line line' of A'B' based on the A' and B' surfaces.
[0064] Step 2304: Use line' to find the intersection point p' with surface C'.
[0065] Step 2305: p-p' is the position offset vector.
[0066] On the other hand, the method of determining the rotation offset between the physical target and the virtual three-dimensional model based on the three planes of the physical target and the three planes of the virtual three-dimensional model includes: constructing a first rotation matrix based on the first normal of the first plane, the second normal of the second plane and the third normal of the third plane; constructing a second rotation matrix based on the fourth normal of the fourth plane, the fifth normal of the fifth plane and the sixth normal of the sixth plane; multiplying the second rotation matrix by the inverse matrix of the first rotation matrix to obtain a combined rotation matrix; converting the combined rotation matrix into a quaternion to obtain a rotation difference quaternion; and using the rotation difference quaternion as the rotation offset.
[0067] Specifically, the rotation offset between the physical object and the virtual three-dimensional model may be determined through the following steps 2306-2309.
[0068] Step 2306: Construct a rotation matrix M according to the normal vectors of the three faces A, B, and C (v1, v2, and v3, respectively), as follows:
[0069]
[0070] Step 2307: Construct a rotation matrix M' according to the normal vectors of the three faces A', B', and C' (v'1, v'2, and v'3, respectively), as follows:
[0071]
[0072] Step 2308: Calculate the combined rotation matrix by multiplying M' by the inverse matrix of M, and then convert it into a quaternion to obtain the rotation difference quaternion R, as follows:
[0073] R = Quaternion (M′×M -1 )
[0074] Step 2309: Use R as the rotation offset.
[0075] Step 240: Determine the target position of the physical object according to the initial position and the position offset.
[0076] Specifically, the initial position is added to the position offset to obtain the target position, as follows:
[0077] TargetPos it ion=OriginPos it ion+p-p'
[0078] Step 250: Determine a target rotation angle of the physical object according to the initial rotation angle and the rotation offset.
[0079] Specifically, the initial rotation angle is multiplied by the rotation offset to obtain the target rotation angle, as follows:
[0080] TargetRotation=R×OriginRotation
[0081] Step 260: Update the position of the virtual three-dimensional model to the target position, and update the rotation angle of the virtual three-dimensional model to the target rotation angle to complete the three-dimensional registration of augmented reality.
[0082] The plane generation method involved in the above steps is the key point for generating the plane required for three-dimensional registration in the embodiment of the present application, which can be implemented using a plane generation algorithm, including: according to the first triangular mesh selected by the user, retrieving multiple second triangular meshes adjacent to the first triangular mesh; when all triangular meshes are close to coplanarity, determining the height of the center point of each second triangular mesh from the first triangular mesh, and obtaining the average height; creating a plane parallel to the first triangular mesh in the normal direction of the center point of the first triangular mesh and at the average height. Wherein, all triangular meshes are close to coplanarity, including: obtaining the vertices of all triangular meshes, wherein each triangular mesh includes three vertices; determining the normalized normal vector of the first triangular mesh according to the three vertices of the first triangular mesh; selecting any one of the three vertices of the first triangular mesh, and forming multiple first vectors with the vertices of the multiple second triangular meshes; judging whether all triangular meshes are close to coplanarity according to the normalized normal vector and the multiple first vectors; when the normalized normal vector and the multiple first vectors meet the preset conditions, determining that all triangular meshes are close to coplanarity.
[0083] Regarding judging whether all the triangular meshes are close to being coplanar based on the normalized normal vector and multiple first vectors; when the normalized normal vector and multiple first vectors satisfy a preset condition, determining that all the triangular meshes are close to being coplanar, specifically includes: normalizing multiple first vectors to obtain multiple normalized first vectors; multiplying the multiple normalized first vectors with the normalized normal vector in turn and taking the absolute values; if these values all satisfy the preset conditions, then it can be determined that all the triangular meshes are close to being coplanar.
[0084] Specifically, when the user selects a virtual 3D model in the augmented reality screen, a physical world plane identified by the augmented reality plane detection, or a 3D mesh of the physical world modeled by augmented reality, a triangular mesh Tri1 will be selected. Based on Tri1, multiple adjacent triangular meshes Tri 2~n , and judge Tri 1~nWhether to approach coplanarity. If it approaches coplanarity, calculate Tri 2~n The arithmetic mean h of the center point from Tri1 (i.e. the average height, first calculate the height of each center point from Tri1, and then calculate the arithmetic mean of all heights to obtain h), create a plane parallel to Tri1 at the height h in the normal direction of the center point of Tri1, that is, the required plane.
[0085] The above judgment Tri 1~n The specific method of approaching coplanarity is as follows:
[0086] Define the triangular mesh Tri 1~n All vertices on it are p1, p2, p3, ... n , the error coefficient k (empirical parameter) is cos85°, that is, the error is 5°.
[0087] If n=3, they are directly considered to be coplanar, and if n≥4, subsequent calculations are performed.
[0088] The three points p1, p2, and p3 form a plane (that is, the first triangular mesh Tri1 in the above embodiment), and the normalized vector of its normal vector (that is, the normalized normal vector of Tri1) is specifically:
[0089]
[0090] p4, p5...p n is a plurality of second triangular meshes Tri in the above embodiment 2~n The vertex, that is, from p4 to p n Calculate in sequence To determine Tri 1~n Whether to approach coplanarity.
[0091] If all It is believed that Tri 1~n Approximately coplanar, otherwise it is considered that Tri 1~n No approximation to coplanarity.
[0092] The above embodiments can effectively improve the cumbersome and high-cost operation of augmented reality virtual-real registration technology in the industrial field, and can quickly register virtual models to physical objects in a new site. It only needs to perform SLAM real-time scanning of the environment and then a few clicks, without the need to manually adjust the position offset and rotation offset of the three axes, nor rely on complex preliminary preparations such as physical object recognition, contour recognition, and site cloud scanning.
[0093] In several embodiments provided in the present application, the disclosed devices and methods can also be implemented in other ways. The device and method embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and a part of a module, a program segment or a code contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of boxes in the block diagram and / or the flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.
[0094] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.
[0095] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program codes.
Claims
1. A method for augmented reality three-dimensional registration, It is characterized in that The following steps are involved: Obtaining the initial position and initial rotation angle of the virtual three-dimensional model; Determine three planes of the physical object and three planes of the virtual three-dimensional model according to user selection and plane generation method; Determining a position offset and a rotation offset between the physical object and the virtual three-dimensional model according to three planes of the physical object and three planes of the virtual three-dimensional model; Determining the target position of the physical object according to the initial position and the position offset; Determining a target rotation angle of the physical object according to the initial rotation angle and the rotation offset; The position of the virtual three-dimensional model is updated to the target position, and the rotation angle of the virtual three-dimensional model is updated to the target rotation angle to complete the three-dimensional registration of augmented reality.
2. The method for augmented reality three-dimensional registration according to claim 1, It is characterized in that The three planes of the physical object are determined according to the user selection and the plane generation method, including: According to the physical world plane or the physical world three-dimensional grid surface selected by the user, a first plane, a second plane and a third plane of the physical object are obtained by using a plane generation method; The first plane, the second plane and the third plane intersect each other and intersect at one point.
3. The method for augmented reality three-dimensional registration according to claim 2, It is characterized in that Determining three planes of the virtual three-dimensional model according to the user selection and the plane generation method includes: According to the planes on the virtual three-dimensional model corresponding to the first plane, the second plane and the third plane selected by the user, a fourth plane, a fifth plane and a sixth plane of the virtual three-dimensional model are obtained by using a plane generation method; The fourth plane, the fifth plane and the sixth plane intersect each other and intersect at one point.
4. The method for augmented reality three-dimensional registration according to claim 2, It is characterized in that The determining of the position offset between the physical object and the virtual three-dimensional model according to the three planes of the physical object and the three planes of the virtual three-dimensional model comprises: determining a first intersection line between the first plane and the second plane; determining a first intersection point between the third plane and the first intersection line; determining a second intersection line between the fourth plane and the fifth plane; determining a second intersection point between the sixth plane and the second intersection line; The position offset is determined according to the first intersection point and the second intersection point.
5. The method for augmented reality three-dimensional registration according to claim 3, It is characterized in that Determining the rotation offset between the physical object and the virtual three-dimensional model according to the three planes of the physical object and the three planes of the virtual three-dimensional model includes: constructing a first rotation matrix according to a first normal of the first plane, a second normal of the second plane, and a third normal of the third plane; constructing a second rotation matrix according to the fourth normal of the fourth plane, the fifth normal of the fifth plane, and the sixth normal of the sixth plane; Multiply the second rotation matrix by the inverse matrix of the first rotation matrix to obtain a combined rotation matrix; Convert the combined rotation matrix into a quaternion to obtain a rotation difference quaternion; The rotation difference quaternion is used as the rotation offset.
6. The method for augmented reality three-dimensional registration according to claim 1, It is characterized in that Determining the target position of the physical object according to the initial position and the position offset includes: The initial position is added to the position offset to obtain the target position.
7. The method for augmented reality three-dimensional registration according to claim 1, It is characterized in that Determining a target rotation angle of the physical object according to the initial rotation angle and the rotation offset includes: The target rotation angle is obtained by multiplying the initial rotation angle by the rotation offset.
8. The method for augmented reality three-dimensional registration according to claim 1, It is characterized in that The plane generation method includes: According to the first triangular surface mesh selected by the user, a plurality of second triangular surface meshes adjacent to the first triangular surface mesh are retrieved; When all triangular surface meshes are close to being coplanar, determine the height of the center point of each second triangular surface mesh from the first triangular surface mesh, and obtain an average height; A plane parallel to the first triangular surface mesh is created in the normal direction of the center point of the first triangular surface mesh and at the average height.
9. The method for augmented reality three-dimensional registration according to claim 8, It is characterized in that All triangular meshes are approximately coplanar, including: Obtaining vertices of all the triangular meshes, wherein each triangular mesh includes three vertices; Determining a normalized normal vector of the first triangular surface mesh according to three vertices of the first triangular surface mesh; Select any one of the three vertices of the first triangular surface mesh to form a plurality of first vectors with the vertices of the plurality of second triangular surface meshes; Determining whether all the triangular surface meshes are approximately coplanar according to the normalized normal vector and the plurality of first vectors; When the normalized normal vector and a plurality of first vectors satisfy a preset condition, it is determined that all the triangular surface meshes are approximately coplanar.
10. An electronic device, It is characterized in that The electronic device comprises: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to execute the method of augmented reality three-dimensional registration as described in any one of claims 1-9.
Citation Information
Patent Citations
Method and system for realizing augmented reality
CN107665506A
Augmented reality virtual-real fusion method and device based on space registration
CN113256814A
Geographic entity registration method and program product
CN118394863A
Augmented reality-based part inspection method, device and equipment and storage medium
CN118411401A
System and methods for mating virtual objects to real-world environments
US20200258315A1
Cited By
Display method and device based on augmented reality
CN119762715A
An augmented reality-based presentation method and apparatus
CN119762715B
