Augmented reality (AR) roaming error correction method and device and electronic equipment

Abstract

The invention discloses an augmented reality (AR) roaming error correction method and device and electronic equipment, and belongs to the technical field of virtual reality. The correction method comprises the following steps: acquiring a live-action space image frame captured by an image capture device; identifying at least one first marker in the real scene space image frame; each first Marker is registered with a second Marker corresponding to the first Marker, a correction matrix of the AR roaming error is obtained, the second Marker is used for representing the Marker with the same coordinate position as the first Marker in the virtual model, and the live-action space is mapped with the virtual model. The method can solve the problems of low AR roaming error correction efficiency, low correction precision or large data processing amount in the prior art.

Description

Technical Field

[0001] The present invention belongs to the technical field of virtual reality, and in particular relates to a correction method, device and electronic equipment for augmented reality (AR) roaming errors. Background Art

[0002] The current nuclear engineering construction pursues safe and efficient development, and puts forward higher requirements and goals for engineering safety, quality, construction period, engineering cost and other aspects.

[0003] With the development of AR (Augmented Reality) technology, its application in the field of architectural design has gradually attracted attention. Generally speaking, under the conditions of good indoor environment and sufficient light, the AR roaming error is as high as 10 cm. Figure 1 As shown in the figure, wall 1 is a concrete wall in the real space, and wall 2 is a wall in the virtual model. In outdoor environments or other harsh conditions such as dim light, AR roaming errors continue to increase, and even model drift occurs, seriously affecting AR business functions. It can be seen that roaming accuracy, as the core of AR technology, is also the key to the success of AR.

[0004] At present, conventional AR roaming error correction methods include manual fine-tuning and point cloud visual map registration. Manual fine-tuning requires complex manual operations and eye observation to achieve error correction. This method is not only time-consuming and labor-intensive, but also easily affected by human factors, and the correction accuracy and consistency are poor. Point cloud visual map registration requires the pre-collection of point cloud data of the test site and saving it to the cloud server. AR collects point cloud data on site and registers it with the cloud data in real time to achieve error correction. However, the construction site environment of nuclear engineering construction is complex and the construction progress is fast. The use of this method requires real-time updating of cloud point cloud data, which requires large data processing volume and complex operation, increasing construction costs, so it cannot be popularized in the field of construction. Therefore, there is an urgent need for a method that can quickly and accurately identify AR roaming errors and perform automatic correction and registration. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide an augmented reality (AR) roaming error correction method, device and electronic device in view of the above-mentioned deficiencies in the prior art, which can quickly and accurately identify AR roaming errors and realize automatic correction and alignment, thereby improving correction efficiency and correction accuracy.

[0006] In a first aspect, the present invention provides a method for correcting augmented reality (AR) roaming errors, comprising: acquiring a real-space image frame captured by an image capture device; identifying at least one first marker in the real-space image frame; aligning each first marker with its corresponding second marker to obtain an AR roaming error correction matrix, wherein the second marker is used to represent a marker in a virtual model with the same coordinate position as the first marker, and the real-space is mapped to the virtual model.

[0007] Preferably, the aligning of each first Marker with its corresponding second Marker to obtain a correction matrix for the AR roaming error specifically includes: obtaining the angular error and the translation error between each first Marker and its corresponding second Marker; and aligning each first Marker with the second Marker according to the angular error and the translation error to obtain a correction matrix for the AR roaming error.

[0008] Preferably, the Marker includes N corner points, where N is greater than 2.

[0009] The method of obtaining the angular error between each first marker and its corresponding second marker specifically includes: calculating the direction vector of each first marker and its corresponding second marker according to the coordinates of the N corner points of the marker; and calculating the angular error between the direction vectors according to the direction vector of each first marker and its corresponding second marker.

[0010] Preferably, the method of obtaining the translation error between each first marker and its corresponding second marker specifically includes: calculating the center coordinates of each first marker and its corresponding second marker based on the coordinates of the N corner points of the marker; calculating the translation error of the X-axis and the Y-axis based on the difference between the center coordinates of each first marker and its corresponding second marker; calculating the side length of each first marker and its corresponding second marker based on the coordinates of the N corner points of the marker; calculating the scaling parameter based on the ratio of the side lengths of each first marker and its corresponding second marker; and obtaining the depth correction distance based on the scaling parameter to obtain the translation error of the Z-axis.

[0011] Preferably, each first marker is aligned with its corresponding second marker according to the angle error and the translation error to obtain a correction matrix for the AR roaming error, specifically including: converting the angle error into Euler angles and calculating the angle error correction matrix RM; calculating the translation error correction matrix TM according to the translation error; and calculating the correction matrix for the AR roaming error based on the angle error correction matrix RM and the angle error correction matrix TM.

[0012] Preferably, after aligning each first marker with its corresponding second marker to obtain the correction matrix of the AR roaming error, the correction method also includes: acquiring the offset index of at least one second marker in the virtual model to obtain the current frame AR roaming error index; judging the difference between the current frame AR roaming error index and a preset threshold; in response to the current frame AR roaming error index being greater than the preset threshold, adjusting the external parameters of the image capture device based on the correction matrix of the AR roaming error.

[0013] Preferably, the offset index of at least one second Marker in the virtual model is obtained to obtain the AR roaming error index of the current frame, specifically including: calculating the offset index of at least one second Marker in the virtual model according to the angular error and translation error between each first Marker and its corresponding second Marker; averaging the offset index of the at least one second Marker to obtain the AR roaming error index of the current frame.

[0014] Preferably, before acquiring the real-space image frame captured by the image capture device, the correction method further comprises: generating a plurality of markers, each marker comprising a marker ID, and setting at least one second marker on a vertical surface of each component in the virtual model.

[0015] In a second aspect, an embodiment of the present invention further provides an augmented reality (AR) roaming error correction device, comprising:

[0016] The acquisition module is used to acquire the real-scene space image frame captured by the image capture device.

[0017] The recognition module is connected to the acquisition module and is used to recognize at least one first marker in the real scene space image frame.

[0018] The correction module is connected to the recognition module and is used to align each first marker with its corresponding second marker to obtain a correction matrix for the AR roaming error, wherein the second marker is used to represent the marker with the same coordinate position as the first marker in the virtual model, and the real scene space is mapped to the virtual model.

[0019] In a third aspect, the present invention further provides an electronic device, comprising: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the method for correcting augmented reality (AR) roaming errors described in the first aspect by executing the computer instructions.

[0020] The method, device and electronic device for correcting the AR roaming error provided by the present invention obtain the correction matrix of the AR roaming error by identifying the first marker in the real-life space image frame and aligning it with the second marker in the virtual model, which is used to correct the AR roaming error. Compared with manual fine-tuning, this correction process realizes automatic identification of roaming errors, and automatic correction and alignment, thereby improving the correction efficiency and correction accuracy. Since the second marker is a marker in the virtual model and has the same coordinate position as the first marker, the roaming error correction is performed by aligning the mapped markers in the real-life space and the virtual model, which reduces the amount of data processing and improves the correction efficiency compared to the point cloud visual map alignment method. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is a schematic diagram of AR roaming error;

[0022] Figure 2 This is a flow chart of a method for correcting augmented reality (AR) roaming errors according to Embodiment 1 of the present invention;

[0023] Figure 3 This is a schematic diagram of the camera depth error calculation principle of Embodiment 1 of the present invention;

[0024] Figure 4 This is a schematic diagram of a correction result of an AR roaming error according to Embodiment 1 of the present invention;

[0025] Figure 5 This is a flow chart of another method for correcting augmented reality (AR) roaming errors according to Embodiment 1 of the present invention;

[0026] Figure 6 This is a schematic diagram of a Marker in Example 1 of the present invention;

[0027] Figure 7 This is a schematic diagram of a marker registration process according to Example 1 of the present invention.

[0028] Among them, 1-concrete wall of real space; 2-wall of virtual model; 3-first marker; 4-second marker. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the technical solution of the present invention, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0030] It should be understood that the specific embodiments and drawings described herein are only used to explain the present invention rather than to limit the present invention.

[0031] It can be understood that, in the absence of conflict, the various embodiments of the present invention and the various features in the embodiments can be combined with each other.

[0032] It can be understood that, for the convenience of description, the drawings of the present invention only show the parts related to the present invention, while the parts irrelevant to the present invention are not shown in the drawings.

[0033] It can be understood that each unit and module involved in the embodiments of the present invention may correspond to only one physical structure, or may be composed of multiple physical structures, or multiple units and modules may be integrated into one physical structure.

[0034] It can be understood that, without conflict, the functions and steps marked in the flowcharts and block diagrams of the present invention may occur in an order different from that marked in the drawings.

[0035] It is understood that the flowcharts and block diagrams of the present invention illustrate the possible architectures, functions, and operations of the systems, devices, equipment, and methods according to the various embodiments of the present invention. Each box in the flowchart or block diagram may represent a unit, module, program segment, or code, which contains executable instructions for implementing the specified functions. Moreover, each box or combination of boxes in the block diagram and flowchart may be implemented by a hardware-based system that implements the specified functions, or may be implemented by a combination of hardware and computer instructions.

[0036] It can be understood that the units and modules involved in the embodiments of the present invention can be implemented by software or hardware. For example, the units and modules can be located in a processor.

[0037] In some related technologies, since nuclear engineering design and construction is a highly dynamic process, the design, construction and supervision links are complex and changeable, and there are often problems such as information asymmetry and poor communication, which leads to low efficiency of project site construction. Traditional methods mainly rely on two-dimensional drawings and models, which are difficult to intuitively display design details and construction effects.

[0038] The use of AR technology in the design and construction of nuclear engineering civil engineering can convert 3D design data into 3D visualization models in real time, and display them at the construction site of nuclear power projects through AR equipment, so as to achieve seamless connection and efficient collaboration between design and construction. Therefore, this embodiment is applied to the design and construction stage of nuclear engineering civil engineering, and is suitable for the complex and numerous items on the site of nuclear power projects. By automatically correcting AR roaming errors and improving projection accuracy, it supports the AR mode to complete high-quality business functions such as visualization technology disclosure, process control, and quality acceptance.

[0039] Embodiment 1:

[0040] In the first aspect, this embodiment provides an augmented reality (AR) roaming error correction method, which can be applied to a server, which can be deployed in a data center or in the cloud, and the server is connected to an image capture device for communication. It can also be applied to a terminal with an image capture device, such as a smart phone, a tablet computer, an AR wearable device, etc.

[0041] Reference Figure 1 The method for correcting the AR roaming error of this embodiment includes:

[0042] Step 101: Acquire a real-world space image frame captured by an image capture device.

[0043] Step 102: Identify at least one first marker in the real-space image frame.

[0044] Step 103, align each first marker with its corresponding second marker to obtain a correction matrix of AR roaming error, wherein the second marker is used to represent the marker with the same coordinate position as the first marker in the virtual model, and the real scene space is mapped to the virtual model.

[0045] Among them, the image capture device is such as a camera, a camera, or a camera array. Marker is a marker, which refers to a pre-set marker in AR technology. It can be a 2D plane object or a 3D object. Examples of markers include: QR codes, specific graphics, patterns, or symbols. A real-space image frame includes a first marker or multiple first markers, such as Figure 1 There are two first markers on the concrete wall shown. The first marker corresponds to the second marker one by one, and both are at the same coordinate position in their respective coordinate systems. It can be understood that the first marker is mapped to the second marker, that is, corresponding markers are set at the same position of the same component in the real scene space and the virtual model respectively.

[0046] In some embodiments, the calibration method further includes: constructing a corresponding relationship between the first Marker and the second Marker based on a Marker ID (Identification).

[0047] In this embodiment, the first marker in the real-life space image frame is identified and aligned with the second marker in the virtual model to obtain a correction matrix for the AR roaming error, which is used to correct the AR roaming error. Compared with manual fine-tuning, this correction process realizes automatic identification of roaming errors, and automatic correction and alignment, thereby improving correction efficiency and accuracy. Since the second marker is a marker in the virtual model and has the same coordinate position as the first marker, roaming error correction is performed by aligning the mapped markers in the real-life space and the virtual model, which reduces the amount of data processing and improves correction efficiency compared to the point cloud visual map alignment method.

[0048] In some embodiments, registering each first marker with its corresponding second marker to obtain a correction matrix for the AR roaming error specifically includes:

[0049] Step 1021: Obtain the angle error and translation error between each first marker and its corresponding second marker.

[0050] Step 1022: align each first marker with the second marker according to the angle error and the translation error to obtain a correction matrix for the AR roaming error.

[0051] In some embodiments, when a real-space image frame includes only one first marker, the first marker and the second marker are aligned according to an angle error and a translation error between the first marker and its corresponding second marker to obtain a correction matrix for AR roaming errors.

[0052] In this embodiment, a correction matrix of AR roaming error is obtained through a first marker in a real-space image frame, which can significantly reduce the workload of data processing and improve the correction efficiency.

[0053] In some embodiments, when a real-space image frame includes multiple first Markers, that is, the mapped virtual model includes multiple second Markers, such as Figure 1There are 2 first markers and 2 corresponding second markers as shown. Each first marker is registered with the second marker according to the angle error and translation error, and the correction matrices corresponding to multiple second markers (or first markers) (such as 2 correction matrices) are obtained. The correction matrices corresponding to multiple second markers (or first markers) are averaged to obtain the correction matrix of the AR roaming error of the current frame.

[0054] In this embodiment, multiple Markers are included in one real-space image frame, and the correction matrix of AR roaming errors is obtained by registering multiple Markers, which can improve the correction accuracy. And because there are many Markers, it can be ensured that each frame includes Markers, so that the AR roaming error can be corrected in real time during the movement of the image capture device. Whether walking, moving or turning, the user can see the real-time adaptation of the virtual model and the real-space, avoiding the discomfort caused by AR roaming errors and improving the user experience.

[0055] In some embodiments, the Marker includes N corner points, where N is greater than 2. The Marker can be a triangular, quadrilateral, or hexagonal marker. When the Marker is a quadrilateral marker (such as a QR code), it has 4 corner points, and each corner point has corresponding corner point coordinates. For example, a rectangular coordinate system is established in the real scene space and the virtual model with the lower left corner point of the component as the coordinate origin, and the Markers in the real scene space and the virtual model have corresponding corner point coordinates.

[0056] In some embodiments, the angular error between each first marker and its corresponding second marker is obtained, specifically including: calculating the direction vector of each first marker and its corresponding second marker based on the coordinates of the N corner points of the marker; and calculating the angular error between the direction vectors based on the direction vector of each first marker and its corresponding second marker.

[0057] In this embodiment, the direction vectors MNx, MNy and MNz of the second Marker in the virtual model are calculated according to the corner coordinates of the Marker; similarly, the direction vectors RNx, RNy and RNz of the first Marker in the real scene space are calculated, and the angular error between the virtual model and the real scene space coordinate system is calculated according to the inverse trigonometric function. It can be understood that for each first Marker in the real scene space (or each second Marker in the virtual model), there is a corresponding angular error result.

[0058] In some embodiments, the translation error between each first marker and its corresponding second marker is obtained, specifically including: calculating the center coordinates of each first marker and its corresponding second marker based on the coordinates of the N corner points of the marker; calculating the translation error of the X-axis and the Y-axis based on the difference between the center coordinates of each first marker and its corresponding second marker; calculating the side length of each first marker and its corresponding second marker based on the coordinates of the N corner points of the marker; calculating the scaling parameter based on the ratio of the side lengths of each first marker and its corresponding second marker; and obtaining the depth correction distance based on the scaling parameter to obtain the translation error of the Z-axis.

[0059] In this embodiment, according to the coordinates of the corner points of the Markers, the center coordinates Mc of the second Marker in the virtual model and the center coordinates Rc of the first Marker in the real scene space are calculated, and the translation error between the X-axis and the Y-axis is calculated according to the difference between the center coordinates of the two. According to the coordinates of the corner points of the Markers, the side length Ml of the second Marker in the virtual model and the side length Rl of the first Marker in the real scene space are calculated, as shown in FIG. Figure 3 As shown, according to the camera imaging and similar triangle principle, the parameter radio of M1 is calculated to scale proportionally to M2, the near plane of the camera is obtained, and the camera depth position is updated according to the proportional scaling parameter radio. Figure 3 It can be seen that Tz is the depth correction distance, that is, the translation error of the Z axis. It can be understood that for each first Marker in the real scene space (or each second Marker in the virtual model), there is a corresponding translation error result.

[0060] In some embodiments, each first marker is aligned with its corresponding second marker according to the angle error and the translation error to obtain a correction matrix for the AR roaming error, specifically including: converting the angle error into Euler angles and calculating the angle error correction matrix RM; calculating the translation error correction matrix TM according to the translation error; and calculating the correction matrix for the AR roaming error based on the angle error correction matrix RM and the angle error correction matrix TM.

[0061] In this embodiment, based on the angle error correction matrix RM and the angle error correction matrix TM, the correction matrix of the AR roaming error is calculated specifically including: according to the right-handed coordinate system, with the current image capture device coordinate system as a reference, the angle error correction matrix RM is right-multiplied by the translation error correction matrix TM to calculate the correction matrix CM. It can be understood that for each first Marker in the real scene space (or each second Marker in the virtual model), there is a corresponding angle error correction matrix RM and translation error correction matrix TM. Therefore, the corresponding correction matrix is ​​calculated based on the angle error correction matrix RM and the translation error correction matrix TM. When the real scene space image frame includes a first Marker, the corresponding correction matrix calculated is the correction matrix of the AR roaming error of the current frame. When the real scene space image frame includes multiple first Markers, the multiple corresponding correction matrices can be averaged to obtain the correction matrix of the AR roaming error of the current frame.

[0062] In some embodiments, after aligning each first marker with its corresponding second marker to obtain a correction matrix of the AR roaming error, the correction method further includes: acquiring an offset index of at least one second marker in the virtual model to obtain an AR roaming error index of a current frame; determining a difference between the current frame AR roaming error index and a preset threshold; in response to the current frame AR roaming error index being greater than the preset threshold, adjusting the external parameters of the image capture device based on the correction matrix of the AR roaming error.

[0063] In this embodiment, the image capture device has the ability to automatically adjust external parameters. The AR roaming error index of the current frame is obtained. If the AR roaming error index exceeds the preset threshold, it indicates that the AR roaming error is large. According to the transformation matrix of the current frame image capture device, the correction matrix of the AR roaming error is multiplied on the right to achieve marker alignment and automatic error correction. It should be noted that the correction matrix CM is multiplied on the right, indicating that the current image capture device coordinate system is used as a reference to perform rotation and translation to achieve automatic correction of the position error of the image capture device. The correction result is shown in the figure below. Figure 4 As shown. Among them, the transformation matrix is ​​a matrix used to characterize the current posture state of the image capture device. In response to the current frame AR roaming error index being less than or equal to the preset threshold, the external parameters of the image capture device are not adjusted. By setting the size comparison between the AR roaming error index and the preset threshold, the correction workload can be appropriately reduced.

[0064] In some embodiments, the offset index of at least one second marker in the virtual model is obtained to obtain the AR roaming error index of the current frame, specifically including: calculating the offset index of at least one second marker in the virtual model according to the angular error and translation error between each first marker and its corresponding second marker; averaging the offset index of at least one second marker to obtain the AR roaming error index of the current frame.

[0065] In this embodiment, according to the correspondence between the first marker and the second marker, the coordinates of the N corner points of the marker are counted in turn, the corner point offset distance (i.e., the angle error and translation error) is calculated, and the average distance is taken as the offset index of the current second marker; the offset index of all the second markers in the current frame is counted, and based on this, the average index is calculated as the error index of the AR roaming of the current frame.

[0066] In some embodiments, before acquiring the real-space image frame captured by the image capture device, the correction method further includes: generating a plurality of markers, each marker including a marker ID, and setting at least one second marker on a vertical surface of each component in the virtual model.

[0067] In this embodiment, the vertical surface of a component in the virtual model includes a second Marker or multiple second Markers. The second Marker can also be set on the horizontal surface or other positions of the component in the virtual model. It should be noted that the first Marker corresponding to the second Marker is also set on the vertical surface of the component in the real space. By setting the Marker on the vertical surface of the component (ie, the vertical surface), when calculating the angle error, there is basically no rotation of 90° around the X-axis. Therefore, the deadlock problem of the Euler angle can be ignored, thereby simplifying the calculation. In addition, it is convenient to manually post Markers in the real space to reduce the influence of manual errors on the accuracy of AR roaming error correction.

[0068] Example 1:

[0069] like Figure 5 As shown, the following is an exemplary description of a correction method for augmented reality AR roaming error in this embodiment. The correction method includes:

[0070] Step 11: Create (generate) multiple Markers and set relevant parameters. This step specifically includes:

[0071] (1) Build the OpenCV environment, configure the Aruco module, and add the Marker creation function aruco::drawMarker().

[0072] (2) Set the marker type and initialize it to cv::aruco::DICT_6X6_250, which means a 6X6 marker (excluding edges).

[0073] (3) Set the unique ID of the Marker, which is an increasing number by default.

[0074] (4) Set the size of the Marker grid. Setting the size will simplify the subsequent calculation of the corner point coordinates.

[0075] (5) Save the Marker and output the image. The Marker diagram is as follows Figure 6 shown.

[0076] Step 12: Paste the same marker at the same coordinate position of the virtual model and the construction site (i.e., the real space). Figure 1 The markers 3 and 4 shown. This step specifically includes:

[0077] (1) Establish a rectangular coordinate system with the lower left corner of the component as the origin, and the lower left corner of the marker as the reference point. Set the number and offset of the markers, set the marker ID, and add the second marker to the virtual model. It should be noted that the left and right edges of the second marker are perpendicular to the horizontal plane, and the upper and lower edges are parallel to the horizontal plane, that is, the second marker is set on the vertical plane of the component.

[0078] (2) At the construction site (i.e., the real-life space), the tape measure is used to measure and locate the lower left corner of the marker. The first marker is added at the same coordinate position as the virtual model. The first marker in the real-life space corresponds to the second marker in the virtual model, and their positions and postures are ensured to be completely consistent.

[0079] Step 13: Count the offset distance of the second marker in the virtual model and update the AR roaming error index. This step specifically includes:

[0080] (1) Build the OpenCV environment, configure the Aruco module, and add the Marker detection function aruco::detectMarkers().

[0081] (2) Identify the second Marker ID of the virtual model, identify the first Marker ID of the real scene space, and establish a mapping relationship between the two Markers.

[0082] (3) Identify the starting point of the marker. In clockwise order, A, B, C, and D are the coordinates of the four corner points of the marker.

[0083] (5) According to the mapping relationship, the coordinates of the four corner points of the marker are counted in turn, the offset distance of the corner points is calculated, and the average distance is taken as the offset index of the current second marker.

[0084] (6) The offset distance of the second marker in the virtual model is counted, and based on this, the average index is calculated as the error index of the AR roaming in the current frame.

[0085] Step 14: Align the markers and calculate the correction matrix. This step specifically includes:

[0086] (1) Based on the corner coordinates of the marker, calculate the direction vectors MNx, MNy and MNz of the second marker in the virtual model; similarly, calculate the direction vectors RNx, RNy and RNz of the first marker in the real space, and calculate the angular error between the virtual model and the real space coordinate system based on the inverse trigonometric function. According to the principle of the virtual model approaching the real space, the angle error correction matrix RM is calculated in the rotation order of the Z-axis-X-axis-Y-axis Euler angle. It is worth mentioning that because the marker is posted on the vertical surface of the component, it will basically not rotate 90° around the X-axis, so the deadlock problem of the Euler angle can be ignored.

[0087] (2) Based on the coordinates of the corner points of the markers, the center coordinates Mc of the second marker in the virtual model and the center coordinates Rc of the first marker in the real space are calculated. Based on the difference between the center coordinates of the two, the offset errors of the X-axis and the Y-axis are calculated to achieve the translation error correction of the XY plane.

[0088] (3) Based on the coordinates of the corner points of the markers, calculate the side length Ml of the second marker in the virtual model and the side length Rl of the first marker in the real scene space, as follows: Figure 3 As shown, according to the camera imaging and similar triangle principle, the parameter radio of M1 is calculated to scale proportionally to M2, the near plane of the camera is obtained, and the camera depth position is updated according to the proportional scaling parameter radio. Figure 3 As can be seen from the above, Tz is the depth correction distance. According to the correction distances of the camera's X-axis, Y-axis, and Z-axis, the translation error correction matrix TM is calculated.

[0089] (4) Based on the right-handed coordinate system, taking the current camera coordinate system as the reference, the angle error correction matrix RM is right-multiplied by the translation error correction matrix TM to calculate the correction matrix CM of the camera extrinsic parameters.

[0090] Step 15: Based on the correction matrix, update the camera extrinsic parameters to realize automatic correction of AR roaming errors. This step specifically includes: obtaining the AR roaming error index of the current frame. If the error index exceeds the preset threshold, it indicates that the AR error is large. According to the transformation matrix of the current frame camera, right-multiply the correction matrix CM of the camera extrinsic parameters to realize marker alignment and error correction. It should be noted that right-multiplying the correction matrix CM indicates that the current camera coordinate system is used as a reference to perform rotation and translation to realize the correction of the camera position error.

[0091] Embodiment 2:

[0092] This embodiment provides an augmented reality (AR) roaming error correction device, including:

[0093] The acquisition module is used to acquire the real-scene space image frame captured by the image capture device.

[0094] The recognition module is connected to the acquisition module and is used to recognize at least one first marker in the real scene space image frame.

[0095] The correction module is connected to the recognition module and is used to align each first marker with its corresponding second marker to obtain a correction matrix for the AR roaming error, wherein the second marker is used to represent the marker with the same coordinate position as the first marker in the virtual model, and the real scene space is mapped to the virtual model.

[0096] In some embodiments, the correction module is used to obtain the angle error and translation error between each first marker and its corresponding second marker, and align each first marker with the second marker according to the angle error and translation error to obtain a correction matrix of the AR roaming error.

[0097] In some embodiments, the Marker includes N corner points, where N is greater than 2.

[0098] The correction module is specifically used to calculate the direction vector of each first marker and its corresponding second marker according to the coordinates of the N corner points of the marker, and is also used to calculate the angular error between the direction vectors according to the direction vector of each first marker and its corresponding second marker.

[0099] In some embodiments, the correction module is specifically used to calculate the center coordinates of each first marker and its corresponding second marker based on the coordinates of the N corner points of the marker, and calculate the translation error of the X-axis and the Y-axis based on the difference between the center coordinates of each first marker and its corresponding second marker. It is also used to calculate the side length of each first marker and its corresponding second marker based on the coordinates of the N corner points of the marker, calculate the scaling parameter based on the ratio of the side lengths of each first marker and its corresponding second marker, and obtain the depth correction distance based on the scaling parameter to obtain the translation error of the Z-axis.

[0100] In some embodiments, the correction matrix is ​​specifically used to convert the angle error into Euler angles, and calculate the angle error correction matrix RM, and calculate the translation error correction matrix TM based on the translation error, and to calculate the correction matrix of AR roaming error based on the angle error correction matrix RM and the angle error correction matrix TM.

[0101] In some embodiments, the calibration device further comprises an adjustment module.

[0102] An adjustment module is used to obtain the offset index of at least one second marker in the virtual model, obtain the current frame AR roaming error index, and determine the magnitude of the current frame AR roaming error index and a preset threshold, and, in response to the current frame AR roaming error index being greater than the preset threshold, adjust the external parameters of the image capture device based on the correction matrix of the AR roaming error.

[0103] In some embodiments, the adjustment module is specifically used to calculate the offset index of at least one second marker in the virtual model based on the angular error and translation error between each first marker and its corresponding second marker, and to average the offset index of at least one second marker to obtain the AR roaming error index of the current frame.

[0104] In some embodiments, the correction device also includes a generation module.

[0105] The generating module is used to generate a plurality of markers, wherein the markers include marker IDs, and is also used to set at least one second marker on a vertical surface of each component in the virtual model.

[0106] Embodiment 3:

[0107] This embodiment provides an electronic device, including: a memory and a processor, the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the augmented reality (AR) roaming error correction method described in Example 1 by executing the computer instructions.

[0108] It is to be understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of the present invention, but the present invention is not limited thereto. For those of ordinary skill in the art, various modifications and improvements can be made without departing from the spirit and essence of the present invention, and these modifications and improvements are also considered to be within the scope of protection of the present invention.

Claims

1. A method for correcting augmented reality (AR) roaming errors, characterized in that: include: Acquire a real-space image frame captured by an image capture device; Identify at least one first marker Marker in the real-space image frame; Each first marker is registered with its corresponding second marker to obtain a correction matrix for the AR roaming error, wherein the second marker is used to represent the marker with the same coordinate position as the first marker in the virtual model, and the real scene space is mapped to the virtual model.

2. The method according to claim 1, characterized in that The registering each first marker with its corresponding second marker to obtain a correction matrix of the AR roaming error specifically includes: Get the angle error and translation error between each first marker and its corresponding second marker; Each first marker is aligned with the second marker according to the angle error and the translation error to obtain a correction matrix of the AR roaming error.

3. The method according to claim 2, characterized in that Marker includes N corner points, N is greater than 2, The obtaining of the angle error between each first marker and its corresponding second marker specifically includes: According to the coordinates of the N corner points of the Marker, the direction vector of each first Marker and its corresponding second Marker is calculated respectively; According to the direction vectors of each first marker and its corresponding second marker, the angle error between the direction vectors is calculated.

4. The method according to claim 3, characterized in that The obtaining of the translation error between each first marker and its corresponding second marker specifically includes: According to the coordinates of the N corner points of the Marker, calculate the center coordinates of each first Marker and its corresponding second Marker respectively; The translation error between the X-axis and the Y-axis is calculated based on the difference between the center coordinates of each first marker and its corresponding second marker; According to the coordinates of the N corner points of the Marker, calculate the side length of each first Marker and its corresponding second Marker respectively; Calculate the scaling parameter based on the ratio of the side length of each first marker to its corresponding second marker; Get the depth correction distance according to the scaling parameter and get the translation error of the Z axis.

5. The method according to claim 2, characterized in that: Aligning each first marker with its corresponding second marker according to the angle error and the translation error to obtain a correction matrix for the AR roaming error specifically includes: The angle error is converted into Euler angles, and an angle error correction matrix RM is calculated; Calculating a translation error correction matrix TM according to the translation error; Based on the angle error correction matrix RM and the angle error correction matrix TM, a correction matrix for AR wandering error is calculated.

6. The method according to claim 2, characterized in that After registering each first marker with its corresponding second marker to obtain a correction matrix of the AR roaming error, the method further includes: Obtaining an offset index of at least one second marker in the virtual model to obtain an AR roaming error index of a current frame; Determine the difference between the AR roaming error index of the current frame and the preset threshold; In response to the current frame AR roaming error index being greater than a preset threshold, adjusting the extrinsic parameters of the image capture device based on a correction matrix of the AR roaming error.

7. The method according to claim 6, characterized in that Acquiring the offset index of at least one second marker in the virtual model to obtain the current frame AR roaming error index specifically includes: Calculating an offset index of at least one second Marker in the virtual model according to an angular error and a translation error between each first Marker and its corresponding second Marker; The offset index of the at least one second Marker is averaged to obtain an AR roaming error index of the current frame.

8. The method according to claim 1, characterized in that Before acquiring the real scene space image frame captured by the image capture device, the method further includes: Generate multiple Markers, each of which includes a Marker ID. At least one second marker is set on a vertical surface of each component in the virtual model.

9. An augmented reality (AR) roaming error correction device, characterized in that: include: An acquisition module is used to acquire a real-space image frame captured by an image capture device. an identification module, connected to the acquisition module, for identifying at least one first Marker in the real scene space image frame, The correction module is connected to the recognition module and is used to align each first marker with its corresponding second marker to obtain a correction matrix for the AR roaming error, wherein the second marker is used to represent the marker with the same coordinate position as the first marker in the virtual model, and the real scene space is mapped to the virtual model.

10. An electronic device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the method for correcting augmented reality (AR) roaming errors according to any one of claims 1 to 8 by executing the computer instructions.

Images