Image display method, device, computer equipment and storage medium

Abstract

The present application discloses an image display method, device, computer equipment and storage medium, which belongs to the field of intelligent terminal technology. Executed by an augmented reality AR device, the method includes: obtaining a homography matrix between the first target plane and the second target plane according to the first target plane and the second target plane; obtaining a target displacement according to the homography matrix and the posture of the inertial measurement unit; obtaining a target posture according to the target displacement, the target posture being the position and posture of the camera coordinate system of the current frame image in the world coordinate system; and displaying an AR image according to the target posture. The present application does not require the step of translating the initialization of the AR device, and also avoids the AR device from reconstructing a large number of points in three-dimensional space, reducing the amount of calculation of the AR device, and improving the efficiency of AR model positioning in the AR device and the effect of displaying the AR model image.

Description

Technical Field

[0001] The present application relates to the field of smart terminal technology, and in particular to an image display method, device, computer equipment, and storage medium. Background Art

[0002] With the development of the field of intelligent terminal technology, terminals can now provide a variety of image displays. Among them, due to the good visual effects of augmented reality (AR) models, many terminals can display objects through AR models.

[0003] Currently, when displaying an AR model in a terminal, the AR model to be placed is often positioned in three-dimensional space. Among them, the positioning method often adopts the following methods: for example, through the Simultaneous Localization And Mapping (SLAM) method, the Visual Inertial Odometry (VIO) method, etc., to achieve the positioning of the AR model in three-dimensional space.

[0004] Among them, for the above-mentioned solutions, when positioning the AR model in the three-dimensional space, a large number of points need to be reconstructed in the three-dimensional space, resulting in a large amount of calculation for AR model positioning. Summary of the Invention

[0005] The embodiments of the present application provide an image display method, apparatus, computer device, and storage medium. These methods can improve the efficiency of AR model positioning in AR devices and the effect of displaying AR model images. The technical solution is as follows:

[0006] In one aspect, an embodiment of the present application provides an image display method, which is performed by an augmented reality (AR) device, and includes:

[0007] Acquire a homography matrix between a first target plane and a second target plane according to a first target plane and a second target plane, wherein the first target plane is the target plane of the current frame image, and the second target plane is the target plane of the first frame image;

[0008] Obtaining a target displacement according to the homography matrix and the inertial measurement unit posture, where the target displacement is a displacement of a camera coordinate system of the current frame image relative to the camera coordinate system of the first frame image;

[0009] Acquire a target pose according to the target displacement, where the target pose is the position and posture of the camera coordinate system of the current frame image in the world coordinate system;

[0010] The AR image is displayed according to the target posture.

[0011] On the other hand, an embodiment of the present application further provides an image display device, which is used in an augmented reality (AR) device, and includes:

[0012] a matrix acquisition module, configured to acquire a homography matrix between a first target plane and a second target plane according to a first target plane and a second target plane, wherein the first target plane is the target plane of the current frame image and the second target plane is the target plane of the first frame image;

[0013] a displacement acquisition module, configured to acquire a target displacement according to the homography matrix and the inertial measurement unit posture, wherein the target displacement is a displacement of the camera coordinate system of the current frame image relative to the camera coordinate system of the first frame image;

[0014] A posture acquisition module, configured to acquire a target posture according to the target displacement, wherein the target posture is the position and posture of the camera coordinate system of the current frame image in the world coordinate system;

[0015] The image display module is used to display the AR image according to the target posture.

[0016] On the other hand, an embodiment of the present application also provides a computer device, which includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the image display method as described above.

[0017] On the other hand, an embodiment of the present application also provides a computer-readable storage medium, which stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement the image display method as described above.

[0018] In one aspect, embodiments of the present application provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the image display method provided in the above-mentioned aspect.

[0019] The beneficial effects of the technical solutions provided in the embodiments of the present application include at least:

[0020] In this application, an AR device obtains a homography matrix between the target plane of the current frame image and the target plane of the first frame image based on the AR device, and obtains the target displacement from the homography matrix in combination with the posture of the inertial measurement unit. The target displacement is the displacement of the camera coordinate system of the current frame image relative to the camera coordinate system of the first frame image; the AR device obtains the position and posture of the camera coordinate system of the current frame image in the world coordinate system based on the target displacement, and displays the AR image according to the obtained position and posture. There is no need to translate the AR device initialization step, and it also avoids the AR device from reconstructing a large number of points in three-dimensional space, reducing the calculation amount of the AR device, and improving the efficiency of AR model positioning in the AR device and the effect of displaying the AR model image. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0022] Figure 1 This is a schematic diagram of the structure of a camera-type AR head-mounted device provided in an embodiment of the present application;

[0023] Figure 2 is a method flow chart of an image display method provided by an embodiment of the present application;

[0024] Figure 3 is a method flow chart of an image display method provided by an embodiment of the present application;

[0025] Figures 4 and 5 This is a schematic diagram of an AR image display interface according to an exemplary embodiment of the present application;

[0026] Figure 6 is a structural block diagram of an image display device shown in an exemplary embodiment of the present application;

[0027] Figure 7 It is a structural diagram of a computer device shown in an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0028] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0029] The solution provided by this application can be used in real-life scenarios where network developers deploy network environments. To facilitate understanding, the following first briefly introduces some special terms and application scenarios.

[0030] 1) Augmented Reality (AR): This technology calculates the position and angle of camera images in real time and adds corresponding images. This technology overlays images onto the real world through projection or direct display on the lens' display screen, allowing the user to interact with the displayed images through the device.

[0031] 2) Inertial Measurement Unit (IMU): A device that measures an object's three-axis attitude angle (or angular rate) and acceleration. Typically, an IMU consists of three single-axis accelerometers and three single-axis gyroscopes. The accelerometers detect the object's acceleration signals along three independent axes in the carrier's coordinate system, while the gyroscopes detect the carrier's angular velocity signals relative to the navigation coordinate system. Together, they measure the object's angular velocity and acceleration in three-dimensional space and use this to calculate the object's attitude.

[0032] 3) Posture: It includes position and attitude. Position can represent the position coordinates of the object in the coordinate system, and attitude can represent the direction vector of the object in the coordinate system.

[0033] The solution provided in this application can be used in real-life scenarios where people display AR models through devices in daily life. To facilitate understanding, some of the terms involved in the embodiments of this application are briefly introduced below.

[0034] AR technology, based on projection methods, can include but is not limited to the following three types: projection AR, display AR, and camera AR. Projection AR uses a transparent lens in its projection device, onto which images of the virtual world are projected; display AR uses a transparent display screen, onto which the virtual world is directly displayed; and camera AR, also known as pseudo-AR, uses an image acquisition component in this type of device to capture real-world images and overlay the real and virtual worlds on the device's display screen.

[0035] Please refer to Figure 1 , which shows a schematic diagram of the composition structure of a camera-type AR head-mounted device provided in an embodiment of the present application. Figure 1 As shown, it includes a graphics card 110, a memory 120, a processor 130, an external device 140, a sensor 150, an image acquisition component 160 and a display screen 170.

[0036] Optionally, the AR head-mounted device can scan or photograph scenes in real life through the image acquisition component 160, send the scanned or photographed data to the processor 130, and display it on the display screen 170 after being processed by the processor 130. Among them, the image acquisition component 160 can include a camera, and optionally, the camera can be a depth camera, and the specific number can be specified by the developer. The image data pre-stored in the memory 120 can also be displayed on the display screen 170 after being processed by the processor 130. Furthermore, the image data pre-stored in the memory 120 can be superimposed on the image scanned or photographed by the image acquisition component 160, and displayed together on the display screen 170.

[0037] Optionally, the memory 120 may store various motion image information and required points for various motions, and may also store corresponding user account information, etc. Furthermore, the memory 120 may also be an integrated local storage device or an extended storage device, such as a pluggable memory card, etc., which is not specifically limited in this embodiment of the present application.

[0038] Optionally, the external device 140 may include, but is not limited to, one or a combination of image recognition technology, gesture recognition technology, and eye tracking technology, such as a computer device with image recognition technology, a smart glove with gesture recognition technology, an eye tracker with eye tracking technology, etc.

[0039] Optionally, the sensor 150 may include, but is not limited to, one or more combinations of an accelerometer, a gyroscope, a gravity sensor, a temperature sensor, and an infrared sensor. The sensor 150 can detect the user's movements through induction principles, and the movements can be processed by the entire AR headset and ultimately displayed on the display screen 170 for the user to see.

[0040] Optionally, in the present application, in addition to Figure 1 In addition to the AR head-mounted devices shown, various terminals such as mobile phones, tablets, e-book readers, smart glasses, smart watches, MP3 players (Moving Picture Experts Group Audio Layer III, Moving Picture Experts Group Audio Layer 3), MP4 (Moving Picture Experts Group Audio Layer IV, Moving Picture Experts Group Audio Layer 4) players, notebook computers, laptop computers and desktop computers can also provide AR model display functions, that is, these terminals are also AR devices.

[0041] Currently, several methods are often used to locate AR devices in three-dimensional space: for example, SLAM and VIO methods are used to achieve the positioning of AR models in three-dimensional space. SLAM and VIO technologies not only require a more accurate inertial measurement unit, but also require the reconstruction of point clouds of tens of thousands of points for positioning. This places high demands on the computing performance of AR devices, thus limiting their application. Furthermore, due to theoretical limitations, users are required to translate the AR device to initialize the system. Before placing an AR model in three-dimensional space, users must actively translate the AR device and wait for the AR device to recognize a plane before placing it.

[0042] In order to solve the problems existing in the above-mentioned related technologies, the present application provides an image display method. Figure 2 , which shows a method flow chart of an image display method provided by an embodiment of the present application, which can be used for the above Figure 1 In the AR device shown, Figure 2 As shown, the method may include the following steps:

[0043] Step 201: Acquire a homography matrix between the first target plane and the second target plane according to the first target plane and the second target plane.

[0044] The first target plane is the target plane of the current frame image, and the second target plane is the target plane of the first frame image. Optionally, the target plane is the plane in the three-dimensional space where the AR model is located in the current frame image. The AR device obtains the homography matrix between the two target planes using the same target plane in the current frame image and the first frame image. The first frame image is the image when the AR device first displays the AR model, and the current frame image is the image currently displayed by the AR device.

[0045] Step 202: Obtain target displacement based on the homography matrix and the inertial measurement unit attitude.

[0046] The IMU pose indicates the direction vector of the current orientation of the IMU coordinate system relative to the original orientation of the IMU coordinate system. The IMU pose includes the IMU pose corresponding to the current frame image and the IMU pose corresponding to the first frame image. The target displacement is the displacement of the camera coordinate system of the current frame image relative to the camera coordinate system of the first frame image. That is, in the first frame image, the IMU in the AR device can correspond to one IMU pose, and in the current frame image, the IMU in the AR device can correspond to a different IMU pose.

[0047] The camera coordinate system is the coordinate system established within the AR device based on the image capture component. For example, if the AR device is a mobile phone, the image capture component can be the phone's camera; if the AR device is a head-mounted device, the image capture component can be the camera that displays the AR model within the AR device. Alternatively, as the AR device moves, the image capture component will also change position, causing the image capture component's coordinate system to change as well. The target displacement can indicate the displacement between the two camera coordinate systems before and after the change.

[0048] Step 203: Obtain the target posture according to the target displacement.

[0049] The target pose is the position and pose of the camera coordinate system of the current frame image in the world coordinate system.

[0050] Optionally, the world coordinate system is a world coordinate system established by the AR device based on the placement position of the AR model in the first frame image and the target plane. That is, the target pose is the position and pose of the camera coordinate system of the current frame image relative to the world coordinate system.

[0051] Step 204: Display the AR image according to the target posture.

[0052] After the AR device obtains the target pose, it displays the corresponding AR image in the current frame image.

[0053] To summarize, the present application obtains the homography matrix between the target plane of the current frame image and the target plane of the first frame image through the AR device, and obtains the target displacement from the homography matrix in combination with the inertial measurement unit posture. The target displacement is the displacement of the camera coordinate system of the current frame image relative to the camera coordinate system of the first frame image; the AR device obtains the position and posture of the camera coordinate system of the current frame image in the world coordinate system based on the target displacement, and displays the AR image according to the obtained position and posture. There is no need to translate the AR device initialization step, and it also avoids the AR device from reconstructing a large number of points in three-dimensional space, reducing the calculation amount of the AR device, and improving the efficiency of AR model positioning in the AR device and the effect of displaying the AR model image.

[0054] In one possible implementation, the AR device obtains the homography matrix between the first target plane and the second target plane based on the first target plane and the second target plane, including the following two homography matrices, wherein the AR device can calculate the first homography matrix between the first target plane and the second target plane according to the first matching algorithm; the AR device can also calculate the second homography matrix between the first target plane and the second target plane according to the target displacement and the inertial measurement unit posture. Taking the homography matrix obtained by the AR device including the first homography matrix and the second homography matrix as an example, for the above Figure 2The illustrated embodiments are presented by way of example.

[0055] Please refer to Figure 3 , which shows a method flow chart of an image display method provided by an embodiment of the present application, which can be used for the above Figure 1 In the AR device shown, Figure 3 As shown, the method may include the following steps:

[0056] Step 301: Determine pixel coordinates in response to a triggering operation in the display screen.

[0057] The trigger operation is used to place the AR model.

[0058] Optionally, in the present application, the AR device may be a terminal including a display screen. The first frame image is the image displayed on the display screen after the AR device places the AR model in the virtual space displayed on the display screen. For example, the user places the AR model in the virtual space by clicking on the display screen of the AR device. At this time, the AR device can obtain the pixel position in the display screen, wherein the coordinates of the pixel position are based on the position coordinates in the pixel coordinate system of the AR device display screen. For example, if the pixel at row 16 and column 18 is triggered, then the position coordinates of the pixel in the pixel coordinate system are the pixel coordinates determined in this step.

[0059] Optionally, in actual applications, when more than one pixel is touched on the display screen, the AR device may determine the position coordinates of the center pixel in the touch area in the pixel coordinate system as the pixel coordinates determined in this step. That is, after the user touches and places the AR model on the display screen, the AR device may obtain the position of the center pixel in the touched area on the display screen and determine the position coordinates of the center pixel in the pixel coordinate system based on the position of the center pixel.

[0060] In one possible implementation, the AR device may also determine the placement position of the AR model based on the pixel coordinates, and use the placement position of the AR model as the origin of the world coordinate system to establish a world coordinate system.

[0061] Optionally, the AR device can obtain a three-dimensional point O on the horizontal plane corresponding to the pixel position in the displayed three-dimensional space based on the pixel position obtained above. The three-dimensional point O is the placement position of the AR model in the three-dimensional space.

[0062] Optionally, after determining the placement position of the AR model as described above, the AR device can establish a three-dimensional world coordinate system in the virtual space based on the placement position of the AR model, with the placement position of the AR model (i.e., the three-dimensional point O mentioned above) as the origin, and the horizontal plane where the placement position of the AR model is located as the plane where the two coordinate axes in the world coordinate system are located.

[0063] Step 302: Calculate the target coordinates based on the pixel coordinates.

[0064] The target coordinates are the coordinates of the AR model's placement relative to the camera coordinate system.

[0065] Optionally, the target coordinates represent the coordinates of the position of the AR model displayed in the three-dimensional space in the camera coordinate system. The target coordinates are represented by P1. The AR device can calculate the target coordinates based on the pixel coordinates according to the following formula [1]:

[0066]

[0067] Where d is a constant value pre-set by the developer, K is the intrinsic matrix of the image capture component of the AR device, n is the normal vector of the horizontal plane in the camera coordinate system, T represents the transpose of n, and P' is the homogeneous coordinate of the touch point coordinate conversion. That is, the AR device can convert the touch point coordinates into the form of homogeneous coordinates P', and substitute P' into formula [1] to obtain P1. Among them, if the coordinates of the touch point P are (u, v), then the homogeneous coordinates P' of P are (u, v, 1).

[0068] In one possible implementation, the intrinsic parameter matrix K of the image capture component may be as follows:

[0069]

[0070] Where f is the focal length of the image capturing component, in pixels. x represents the abscissa of the optical center of the image capturing component in the camera coordinate system, and y represents the ordinate of the optical center of the image capturing component in the camera coordinate system.

[0071] Optionally, the AR device can also obtain the above n through the rotation posture corresponding to the first frame image of the image capturing component. That is, in the above step 301, after the user places the AR model, the AR device can obtain the IMU posture R from the inertial measurement unit of the AR device. IMU1 , where the IMU attitude R IMU1 The direction vector used to indicate the current orientation of the IMU coordinate system relative to the original orientation of the IMU coordinate system. For example, the original orientation of the IMU coordinate system is horizontally facing north. In the first frame image, the current orientation of the IMU coordinate system is in another direction. The IMU attitude R IMU1 The change between the two can be described. The AR device is based on the posture R of the IMU. IMU1 Calculate the rotation posture of the image capturing component under the first frame image, wherein the AR device can R IMU1 Substitute into the following formula [2].

[0072] R1=R -1 ic *R IMU1 *R ic 【2】

[0073] Among them, R ic is the rotational attitude of the camera coordinate system relative to the IMU coordinate system. Optionally, with the first direction as the attitude of the IMU coordinate system, when the image capturing component faces the first direction, the rotational attitude of the image capturing component is expressed by R G It means, then, R G as follows:

[0074]

[0075] Optionally, the first direction may be horizontally toward north.

[0076] The AR device calculates the rotation posture R1 of the image capturing component corresponding to the first frame image through the above formula [2], and substitutes R1 into formula [3].

[0077] n=R1 -1 *(0,0,-1) T 【3】

[0078] The AR device calculates the normal vector n of the horizontal plane using the above formula [3].

[0079] Step 303: Obtain the first pose according to the target coordinates.

[0080] In one possible implementation, the AR device can also obtain a first pose based on the target coordinates obtained above. The first pose is the position and pose of the camera coordinate system of the first frame image in the world coordinate system. When displaying the AR image according to the target pose, the AR image is displayed based on the first pose and the target pose.

[0081] Among them, the first pose can be calculated by substituting the target coordinates into formula [4] to calculate the target pose. W,C1 Represents the first pose, formula [4] is as follows:

[0082]

[0083] Where A = (0, 0, 0).

[0084] Step 304 : Acquire a homography matrix between the first target plane and the second target plane according to the first target plane and the second target plane.

[0085] Among them, the first target plane is the target plane of the current frame image, and the second target plane is the target plane of the first frame image. Optionally, the target plane can be the plane where the above-mentioned AR model is placed. For example, the target plane can be a plane parallel to the horizontal plane where the origin coordinates of the above-mentioned world coordinate system are located. Optionally, the above-mentioned AR device can know the position where the AR model needs to be placed in the three-dimensional space based on the position of the touch point on the display screen, and can display the AR model on the display screen to form the first frame image.

[0086] In one possible implementation, the AR device obtains two homography matrices between the first target plane and the second target plane, one of which (the first homography matrix) is calculated by the AR device according to the first matching algorithm, and the other homography matrix (the second homography matrix) is calculated based on the target displacement and the inertial measurement unit posture, where the target displacement is the displacement of the camera coordinate system of the current frame image relative to the camera coordinate system of the first frame image.

[0087] Optionally, the AR device may calculate a first homography matrix between the first target plane and the second target plane based on a first matching algorithm. The first matching algorithm may be pre-set in the AR device by a developer, and may be any matching algorithm corresponding to a feature point method or a template matching method.

[0088] Optionally, the first homography matrix between the first target plane and the second target plane calculated by the AR device according to the first matching algorithm may be as follows:

[0089]

[0090] Among them, H' 2,1 A first homography matrix between the first target plane and the second target plane may be represented.

[0091] Optionally, to facilitate calculation, the AR device may also divide the values ​​of each element in the first homography matrix obtained above by the value of the element in the last row and last column, and use H 2,1 To express, then, H 2,1 It can be:

[0092]

[0093] After simplification, H 2,1 It is expressed as follows:

[0094]

[0095] In one possible implementation, the AR device may further calculate a second homography matrix between the first target plane and the second target plane according to the target displacement and the inertial measurement unit posture.

[0096] Optionally, the AR device obtains the rotation posture of the image capturing component in the current frame image based on the posture of the inertial measurement unit corresponding to the current frame image, and the rotation posture is used to indicate the posture of the image capturing component relative to the target direction; obtains the rotation posture of the image capturing component in the first frame image based on the posture of the inertial measurement unit corresponding to the first frame image; calculates the normal vector of the target plane based on the rotation posture of the image capturing component in the first frame image; calculates the second homography matrix based on the target displacement, the rotation posture of the image capturing component in the current frame image, the rotation posture of the image capturing component in the first frame image, and the normal vector of the target plane.

[0097] Among them, the AR device obtains the rotational posture of the image capture component under the current frame image based on the posture of the inertial measurement unit corresponding to the current frame image. The above formula [2] can be referred to, in which the AR device obtains the rotational posture of the image capture component under the first frame image based on the posture of the inertial measurement unit corresponding to the first frame image. That is, if R2 is used to represent the rotational posture of the image capture component under the current frame image, under the current frame image, the AR device can also obtain the posture R of the IMU from the inertial measurement unit of the AR device. IMU2 , and according to the attitude R of the IMU IMU2 Calculate the rotation posture of the image capturing component under the current frame image, where the AR device can IMU2 Substitute into the following formula [5].

[0098] R2=R -1 ic *R IMU2 *R ic 【5】

[0099] The AR device obtains the second homography matrix according to formula [6].

[0100] M' 2,1 =K(R2 -1 *R1-d -1 (t 2,1 n T ))K -1 【6】

[0101] Among them, t 2,1 represents the target displacement. Alternatively, if M' is expressed in matrix form 2,1, , then the second homography matrix M' between the first target plane and the second target plane 21 It can be as follows:

[0102]

[0103] Optionally, to facilitate calculation, the AR device may also divide each element in the second homography matrix obtained above by the element value of the last row and last column, and use M 2,1 To express, then, M 2,1 It can be:

[0104]

[0105] After simplification, M 2,1 It is expressed as follows:

[0106]

[0107] Optionally, the AR device may obtain both the first homography matrix and the second homography matrix obtained as homography matrices.

[0108] Step 305: Obtain the target displacement according to the homography matrix and the inertial measurement unit attitude.

[0109] The inertial measurement unit posture includes the inertial measurement unit posture corresponding to the current frame image and the inertial measurement unit posture corresponding to the first frame image, and the target displacement is the displacement of the camera coordinate system of the current frame image relative to the camera coordinate system of the first frame image.

[0110] In one possible implementation, the AR device can establish a target expression based on the first homography matrix and the second homography matrix, where the target expression is used to indicate that the first homography matrix and the second homography matrix are equal; and obtain a target displacement based on a target optimization method and the target expression, where the target optimization method is used to minimize the value of the target expression by adjusting the target displacement.

[0111] For example, according to the first homography matrix H 2,1 and the second homography matrix M 2,1 A target expression F can be established, where F = (h 0,0 -m 0,0 ) 2 +(h 0,1 -m 0,1 ) 2 +(h 0,2 -m 0,2 ) 2 +(h 1,0 -m 1,0 ) 2 +(h 1,1 -m 1,1 ) 2 +(h 1,2 -m 1,2 ) 2 +(h 2,0 -m 2,0 ) 2 +(h2,1 -m 2,1 ) 2 +(h 2,2 -m 2,2 ) 2 Among them, due to h 2,2 =m 2,2 =1, so F can be expressed as F=(h 0,0 -m 0,0 ) 2 +(h 0,1 -m 0,1 ) 2 +(h 0,2 -m 0,2 ) 2 +(h 1,0 -m 1,0 ) 2 +(h 1,1 -m 1,1 ) 2 +(h 1,2 -m 1,2 ) 2 +(h 2,0 -m 2,0 ) 2 +(h 2,1 -m 2,1 ) 2 .

[0112] Optionally, the AR device may adjust the value of t2,1 during the calculation of M2,1 using a target optimization method to minimize F. Optionally, the target optimization method may be any one of a gradient descent method, a Gauss-Newton method, a Levenberg–Marquardt algorithm, and the like.

[0113] Correspondingly, when F is minimum, the AR device can 2,1 The value of is obtained as the target displacement, thereby obtaining the target displacement.

[0114] Step 306: Obtain the target posture according to the target displacement.

[0115] The target pose is the position and pose of the camera coordinate system of the current frame image in the world coordinate system.

[0116] Optionally, the AR device can substitute the obtained target displacement into formula [7] to calculate the target pose. W,C2 represents the target pose, formula [7] is as follows:

[0117]

[0118] Where A = (0, 0, 0).

[0119] Optionally, the current frame image may be the second frame image, the third frame image, the fourth frame image, and so on relative to the first frame image. The AR device may calculate the target pose corresponding to any subsequent frame image by the same method.

[0120] Step 307: Display the AR image according to the first pose and the target pose.

[0121] Optionally, the AR device may display an AR image on a display screen according to the obtained target posture. When displaying the AR image according to the target posture, the position coordinates of the AR model in the world coordinate system remain unchanged and the AR image is displayed.

[0122] Please refer to Figures 4 and 5 , which shows an interface diagram of an AR image display interface involved in an exemplary embodiment of the present application. Figure 4 As shown, the AR device includes an AR model 401, a camera coordinate system 402, and a world coordinate system 403. Figure 5 As shown, the AR device includes an AR model 501, a camera coordinate system 502, and a world coordinate system 503. The user can move the AR device to change the position of the AR device. With respect to the position change of the AR device, the position and posture of the camera coordinate system 402 relative to the world coordinate system 403 also change, but the established world coordinate system 403 remains unchanged. Figure 5 In the , the AR model is still displayed according to its coordinates in the world coordinate system.

[0123] To summarize, the present application obtains the homography matrix between the target plane of the current frame image and the target plane of the first frame image through the AR device, and obtains the target displacement from the homography matrix in combination with the inertial measurement unit posture. The target displacement is the displacement of the camera coordinate system of the current frame image relative to the camera coordinate system of the first frame image; the AR device obtains the position and posture of the camera coordinate system of the current frame image in the world coordinate system based on the target displacement, and displays the AR image according to the obtained position and posture. There is no need to translate the AR device initialization step, and it also avoids the AR device from reconstructing a large number of points in three-dimensional space, reducing the calculation amount of the AR device, and improving the efficiency of AR model positioning in the AR device and the effect of displaying the AR model image.

[0124] The following are device embodiments of the present application, which can be used to implement the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0125] Figure 6 This is a structural block diagram of an image display device shown in an exemplary embodiment of the present application. The image display device can be used in a server to perform Figure 2 or Figure 3 All or part of the steps in the method shown in the corresponding embodiment are performed by the AR device. The image display device may include the following modules:

[0126] A matrix acquisition module 601 is configured to acquire a homography matrix between a first target plane and a second target plane according to the first target plane and the second target plane, wherein the first target plane is the target plane of the current frame image and the second target plane is the target plane of the first frame image;

[0127] a displacement acquisition module 602 for acquiring a target displacement according to the homography matrix and the inertial measurement unit posture, wherein the target displacement is a displacement of the camera coordinate system of the current frame image relative to the camera coordinate system of the first frame image;

[0128] A posture acquisition module 603 is used to acquire a target posture according to the target displacement, where the target posture is the position and posture of the camera coordinate system of the current frame image in the world coordinate system;

[0129] The image display module 604 is configured to display an AR image according to the target posture.

[0130] Optionally, the matrix acquisition module 601 includes: a first calculation unit, a second calculation unit and a first acquisition unit;

[0131] The first calculation unit is configured to calculate a first homography matrix between the first target plane and the second target plane according to a first matching algorithm;

[0132] The second calculation unit is used to calculate a second homography matrix between the first target plane and the second target plane according to the target displacement and the inertial measurement unit posture;

[0133] The first acquiring unit is configured to acquire both the first homography matrix and the second homography matrix as the homography matrix.

[0134] Optionally, the second calculation unit includes: a first acquisition subunit, a second acquisition subunit, a first calculation subunit and a second calculation subunit;

[0135] The first acquisition subunit is used to acquire a rotational posture of the image capturing component corresponding to the current frame image based on the posture of the inertial measurement unit corresponding to the current frame image, wherein the rotational posture is used to indicate the posture of the image capturing component relative to the target direction;

[0136] The second acquisition subunit is configured to acquire a rotational posture of the image capturing component corresponding to the first frame of image based on a posture of the inertial measurement unit corresponding to the first frame of image;

[0137] The first calculation subunit is configured to calculate a normal vector of the target plane according to a rotation posture corresponding to the image capturing component in the first frame of image;

[0138] The second calculation subunit is used to calculate the second homography matrix based on the target displacement, the rotation posture of the image capturing component under the current frame image, the rotation posture of the image capturing component under the first frame image, and the normal vector of the target plane.

[0139] Optionally, the displacement acquisition module 602 includes: a first establishing unit and a second acquiring unit;

[0140] The first establishing unit is configured to establish a target expression according to the first homography matrix and the second homography matrix, wherein the target expression is configured to indicate that the first homography matrix and the second homography matrix are equal;

[0141] The second acquisition unit is used to acquire the target displacement according to a target optimization method and the target expression, wherein the target optimization method is used to minimize the value of the target expression by adjusting the target displacement.

[0142] Optionally, the AR device includes a display screen, and the apparatus further includes:

[0143] a coordinate determination module, configured to determine pixel coordinates in response to a trigger operation on the display screen before the matrix acquisition module 601 acquires a homography matrix between the first target plane and the second target plane based on the first target plane and the second target plane, wherein the trigger operation is used to place the AR model;

[0144] A position determination module, configured to determine a placement position of the AR model based on the pixel coordinates;

[0145] The coordinate system establishment module is used to establish the world coordinate system by taking the placement position of the AR model as the origin of the world coordinate system.

[0146] Optionally, the device further comprises:

[0147] A coordinate calculation module, configured to calculate target coordinates based on the pixel coordinates, where the target coordinates are the coordinates of the placement position of the AR model relative to the camera coordinate system;

[0148] A posture acquisition module, configured to acquire the first posture according to the target coordinates, where the first posture is the position and posture of the camera coordinate system of the first frame image in the world coordinate system;

[0149] The image display module 604 is configured to display the AR image according to the first posture and the target posture.

[0150] Optionally, the inertial measurement unit posture includes the inertial measurement unit posture corresponding to the current frame image and the inertial measurement unit posture corresponding to the first frame image.

[0151] To summarize, the present application obtains the homography matrix between the target plane of the current frame image and the target plane of the first frame image through the AR device, and obtains the target displacement from the homography matrix in combination with the inertial measurement unit posture. The target displacement is the displacement of the camera coordinate system of the current frame image relative to the camera coordinate system of the first frame image; the AR device obtains the position and posture of the camera coordinate system of the current frame image in the world coordinate system based on the target displacement, and displays the AR image according to the obtained position and posture. There is no need to translate the AR device initialization step, and it also avoids the AR device from reconstructing a large number of points in three-dimensional space, reducing the calculation amount of the AR device, and improving the efficiency of AR model positioning in the AR device and the effect of displaying the AR model image.

[0152] It should be noted that the device provided in the above embodiment, when performing the above steps, is illustrated only by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the device provided in the above embodiment and the method embodiment of the above-mentioned image display method are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0153] Figure 7 This is a block diagram of a computer device according to an exemplary embodiment of the present application. Computer device 700 may be a user terminal, such as a smartphone, tablet computer, laptop computer, or desktop computer. Computer device 700 may also be referred to as user equipment, portable terminal, laptop terminal, desktop terminal, or other similar names.

[0154] Typically, the computer device 700 includes a processor 701 and a memory 702 .

[0155] The processor 701 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 701 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), or PLA (Programmable Logic Array). The processor 701 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 701 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 701 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0156] The memory 702 may include one or more computer-readable storage media, which may be non-transitory. The memory 702 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 702 is used to store at least one instruction, which is used to be executed by the processor 701 to implement all or part of the steps performed by the terminal in the message display method provided in the method embodiment of the present application.

[0157] In some embodiments, computer device 700 may optionally include a peripheral device interface 703 and at least one peripheral device. Processor 701, memory 702, and peripheral device interface 703 may be connected via a bus or signal lines. Each peripheral device may be connected to peripheral device interface 703 via a bus, signal lines, or circuit boards. Specifically, the peripheral device may include at least one of a radio frequency circuit 704, a display screen 705, a camera assembly 706, an audio circuit 707, and a power supply 708.

[0158] The peripheral device interface 703 can be used to connect at least one I / O (Input / Output)-related peripheral device to the processor 701 and the memory 702. In some embodiments, the processor 701, the memory 702, and the peripheral device interface 703 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 701, the memory 702, and the peripheral device interface 703 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0159] The RF circuit 704 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 704 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 704 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the RF circuit 704 includes an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The RF circuit 704 can communicate with other terminals via at least one wireless communication protocol. Such wireless communication protocols include, but are not limited to, the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 704 may also include circuitry related to Near Field Communication (NFC), which is not limited in this application.

[0160] Display screen 705 is used to display a user interface (UI). This UI may include graphics, text, icons, videos, or any combination thereof. When display screen 705 is a touchscreen display, it is also capable of collecting touch signals on or above the surface of display screen 705. These touch signals can be input as control signals to processor 701 for processing. In this case, display screen 705 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there can be a single display screen 705, located on the front panel of computer device 700. In other embodiments, there can be at least two display screens 705, located on different surfaces of computer device 700 or in a foldable design. In still other embodiments, display screen 705 can be a flexible display screen, located on a curved or foldable surface of computer device 700. Furthermore, display screen 705 can be configured as a non-rectangular, irregular shape, also known as a special-shaped screen. Display screen 705 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0161] The camera assembly 706 is used to capture images or videos. Optionally, the camera assembly 706 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 706 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0162] The audio circuit 707 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals to be input into the processor 701 for processing, or input into the radio frequency circuit 704 to achieve voice communication. For the purpose of stereo sound collection or noise reduction, there can be multiple microphones, which are respectively arranged in different parts of the computer device 700. The microphone can also be an array microphone or an omnidirectional collection microphone. The speaker is used to convert the electrical signals from the processor 701 or the radio frequency circuit 704 into sound waves. The speaker can be a traditional thin film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signals into sound waves audible to humans, but also convert the electrical signals into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 707 may also include a headphone jack.

[0163] Power supply 708 is used to power various components in computer device 700. Power supply 708 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 708 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, while a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0164] In some embodiments, the computer device 700 further includes one or more sensors 710 , including but not limited to: an acceleration sensor 711 , a gyroscope sensor 712 , a pressure sensor 713 , an optical sensor 714 , and a proximity sensor 715 .

[0165] The accelerometer 711 can detect the magnitude of acceleration along the three coordinate axes of the coordinate system established by the computer device 700. For example, the accelerometer 711 can be used to detect the components of gravity acceleration along the three coordinate axes. The processor 701 can control the display screen 705 to display the user interface in a landscape or portrait view based on the gravity acceleration signal collected by the accelerometer 711. The accelerometer 711 can also be used to collect game or user motion data.

[0166] The gyroscope sensor 712 can detect the orientation and rotation angle of the computer device 700. It can also work with the accelerometer 711 to collect 3D motions of the user on the computer device 700. Based on the data collected by the gyroscope sensor 712, the processor 701 can implement the following functions: motion sensing (e.g., changing the UI based on the user's tilt), image stabilization during shooting, game control, and inertial navigation.

[0167] The pressure sensor 713 can be installed on the side frame of the computer device 700 and / or below the display screen 705. When the pressure sensor 713 is installed on the side frame of the computer device 700, it can detect the user's grip signal of the computer device 700. The processor 701 can perform left and right hand recognition or shortcut operations based on the grip signal collected by the pressure sensor 713. When the pressure sensor 713 is installed below the display screen 705, the processor 701 controls the operational controls on the UI interface based on the user's pressure operation on the display screen 705. The operational controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0168] Optical sensor 714 is used to detect ambient light intensity. In one embodiment, processor 701 can control the display brightness of display screen 705 based on the ambient light intensity detected by optical sensor 714. Specifically, when the ambient light intensity is high, the display brightness of display screen 705 is increased; when the ambient light intensity is low, the display brightness of display screen 705 is decreased. In another embodiment, processor 701 can also dynamically adjust the shooting parameters of camera assembly 706 based on the ambient light intensity detected by optical sensor 714.

[0169] Proximity sensor 715, also known as a distance sensor, is typically located on the front panel of computer device 700. Proximity sensor 715 is used to detect the distance between the user and the front of computer device 700. In one embodiment, when proximity sensor 715 detects that the distance between the user and the front of computer device 700 is gradually decreasing, processor 701 controls display screen 705 to switch from the screen-on state to the screen-off state. When proximity sensor 715 detects that the distance between the user and the front of computer device 700 is gradually increasing, processor 701 controls display screen 705 to switch from the screen-off state to the screen-on state.

[0170] Those skilled in the art will understand that Figure 7 The structure shown in the figure does not constitute a limitation on the computer device 700, and the computer device 700 may include more or fewer components than shown in the figure, or combine some components, or adopt a different component arrangement.

[0171] In an exemplary embodiment, a non-temporary computer-readable storage medium including instructions is also provided, such as a memory including a computer program (instructions), and the above program (instructions) can be executed by a processor of a computer device to complete all or part of the steps of the method shown in each embodiment of the present application. For example, the non-temporary computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a CD-ROM (Compact Disc Read-Only Memory) drive, a magnetic tape, a floppy disk, and an optical data storage device. Optionally, the storage medium stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, at least one program, a code set, or an instruction set is loaded and executed by a processor to implement the image display method involved in the above embodiment.

[0172] The present application also provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the image display method provided in the various optional implementations of the above-mentioned embodiments.

[0173] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0174] It should be understood that the present application is not limited to the exact structures described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. An image display method, characterized in that: The method is performed by an augmented reality (AR) device, and includes: Acquire a homography matrix between a first target plane and a second target plane according to a first target plane and a second target plane, wherein the first target plane is a target plane of a current frame image, and the second target plane is the target plane of a first frame image; Obtaining a target displacement according to the homography matrix and an inertial measurement unit (IMU) posture, wherein the IMU posture is used to indicate a direction vector of a current orientation of an IMU coordinate system relative to an original orientation of the IMU coordinate system, and the target displacement is a displacement of a camera coordinate system of the current frame image relative to the camera coordinate system of the first frame image; Acquire a target pose according to the target displacement, where the target pose is the position and posture of the camera coordinate system of the current frame image in the world coordinate system; Displaying an AR image according to the target posture; The acquiring, based on the first target plane and the second target plane, a homography matrix between the first target plane and the second target plane includes: Calculating a first homography matrix between the first target plane and the second target plane according to a first matching algorithm, wherein the first matching algorithm includes any one of a feature point method and a template matching method; Calculating a second homography matrix between the first target plane and the second target plane according to the target displacement and the IMU posture; The first homography matrix and the second homography matrix are both acquired as the homography matrices.

2. The method according to claim 1, characterized in that The calculating, according to the target displacement and the IMU attitude, a second homography matrix between the first target plane and the second target plane includes: Obtaining, based on the inertial measurement unit posture corresponding to the current frame image, a rotation posture of the image capturing component corresponding to the current frame image, wherein the rotation posture is used to indicate the posture of the image capturing component relative to a target direction; Obtaining a rotational posture of the image capturing component corresponding to the first frame of image based on the posture of the inertial measurement unit corresponding to the first frame of image; Calculating a normal vector of the target plane according to a rotational posture of the image capturing component corresponding to the first frame of image; The second homography matrix is ​​calculated according to the target displacement, the rotation posture of the image capturing component corresponding to the current frame image, the rotation posture of the image capturing component corresponding to the first frame image, and the normal vector of the target plane.

3. The method according to claim 1, characterized in that The step of obtaining the target displacement according to the homography matrix and the inertial measurement unit (IMU) posture includes: Establishing a target expression based on the first homography matrix and the second homography matrix, wherein the target expression is used to indicate that the first homography matrix and the second homography matrix are equal; The target displacement is obtained according to a target optimization method and the target expression. The target optimization method is used to minimize the value of the target expression by adjusting the target displacement.

4. The method according to any one of claims 1 to 3, characterized in that: The AR device includes a display screen, and before acquiring the homography matrix between the first target plane and the second target plane according to the first target plane and the second target plane, further includes: Determining pixel coordinates in response to a trigger operation in the display screen, wherein the trigger operation is used to place an AR model; Determining a placement position of the AR model according to the pixel coordinates; The placement position of the AR model is used as the origin of the world coordinate system to establish the world coordinate system.

5. The method according to claim 4, characterized in that The method further comprises: Calculating target coordinates based on the pixel coordinates, where the target coordinates are coordinates of the placement position of the AR model relative to the camera coordinate system; Acquire a first pose according to the target coordinates, where the first pose is the position and pose of the camera coordinate system of the first frame image in the world coordinate system; The displaying of the AR image according to the target posture includes: The AR image is displayed according to the first posture and the target posture.

6. The method according to any one of claims 1 to 3, characterized in that: The inertial measurement unit posture includes the inertial measurement unit posture corresponding to the current frame image and the inertial measurement unit posture corresponding to the first frame image.

7. An image display device, characterized in that: The device is used in an augmented reality (AR) device, and includes: a matrix acquisition module, configured to acquire a homography matrix between a first target plane and a second target plane according to a first target plane and a second target plane, wherein the first target plane is the target plane of the current frame image and the second target plane is the target plane of the first frame image; a displacement acquisition module, configured to acquire a target displacement according to the homography matrix and an inertial measurement unit (IMU) posture, wherein the target displacement is a displacement of the camera coordinate system of the current frame image relative to the camera coordinate system of the first frame image; A posture acquisition module, configured to acquire a target posture according to the target displacement, wherein the target posture is the position and posture of the camera coordinate system of the current frame image in the world coordinate system; An image display module, configured to display an AR image according to the target posture; Wherein, the matrix acquisition module is further used for: Calculating a first homography matrix between the first target plane and the second target plane according to a first matching algorithm, wherein the first matching algorithm includes any one of a feature point method and a template matching method; Calculating a second homography matrix between the first target plane and the second target plane according to the target displacement and the IMU posture; The first homography matrix and the second homography matrix are both acquired as the homography matrices.

8. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the image display method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the image display method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium. A processor reads and executes the computer instructions from the computer-readable storage medium to implement the image display method according to any one of claims 1 to 6.

Images