Display device and virtual object pose updating method

Abstract

The display device and the virtual object pose updating method provided in the embodiments of the present application can reduce the calculation amount of the front-end tracking module to some extent, and achieve the purpose of outputting the optimized pose of the current frame in real time. Furthermore, after the loop updating of the loop optimization module, the pose of each anchor point in the anchor point frame can be optimized, and when the camera returns to the historical area, the virtual object (anchor point) will still be in the original position, eliminating the drift of the virtual object in the map and improving the user experience of AR / VR.

Description

Technical Field

[0001] This application relates to the field of document display technology, and in particular to a display device and a method for updating the pose of virtual objects. Background Technology

[0002] One of the main applications of AR (Augmented Reality) / VR (Virtual Reality) technology is placing virtual objects in a virtual world. In SLAM (simultaneous localization and mapping), virtual objects are represented by anchor points. The position of the anchor point is the position of the virtual object, and the pose of the anchor point is the pose of the virtual object.

[0003] Because the camera pose can drift during long-term SLAM operation, if we place an anchor point at time t0, the anchor point's position will be offset when we return to the same area at time t1. A common approach is to add a loop closure optimization to eliminate this camera pose (including position and orientation) drift. However, this loop closure optimization is resource-intensive and makes it difficult to calculate the current pose drift in real time. Summary of the Invention

[0004] In order to solve the technical problems existing in the prior art, this application provides a display device and a method for updating the pose of virtual objects.

[0005] The first aspect of this application provides a display device, which includes a front-end tracking module, a back-end optimization module, and a loop closure optimization module.

[0006] The backend optimization module is used to optimize the backend keyframes based on the received IMU data to obtain the optimized pose and velocity of the backend keyframes, wherein the backend keyframes are keyframes of the local map.

[0007] The loop closure optimization module is used to perform SLAM loop closure update on the received camera data to obtain the drift correction amount;

[0008] The front-end tracking module is used to receive IMU data, camera data, optimized pose and velocity of back-end keyframes, and drift correction amount. The camera data includes front-end keyframes.

[0009] If the timestamp of the front-end keyframe is different from the timestamp of the back-end keyframe, then the variables of the front-end keyframe are optimized based on the IMU data, camera data, and drift correction amount.

[0010] If the timestamp of the front-end keyframe is the same as the timestamp of the back-end keyframe, then the optimized pose and velocity of the back-end keyframe are used as the optimized pose and velocity of the front-end keyframe. Based on the camera data, the optimized pose and velocity of the front-end keyframe, the drift correction amount, and other variables of the front-end keyframe are optimized. The other variables are the variables other than pose and velocity.

[0011] In conjunction with the first possible implementation of the first aspect, the front-end tracking module is specifically used for:

[0012] Calculate the inertial navigation pre-integration terms for the previous frame and the current frame, where the current frame is the front-end key frame with the latest timestamp that the front-end tracking module is processing, and the previous frame is the front-end key frame preceding the current frame.

[0013] In conjunction with the second possible implementation of the first aspect, the front-end tracking module is specifically used for:

[0014] The inertial navigation pre-integration term is used to perform inertial navigation calculation on the pose of the previous frame to obtain the initial pose of the current frame.

[0015] In conjunction with the third possible implementation of the first aspect, the front-end tracking module is specifically used for:

[0016] Obtain a 3D point cloud in the local map that matches the feature points of the current frame;

[0017] The initial pose of the current frame is obtained by performing PnP calculation on the 3D point cloud.

[0018] In conjunction with the fourth possible implementation of the first aspect, the front-end tracking module is specifically used for:

[0019] The pose of the previous frame is calculated using the inertial navigation pre-integration term;

[0020] Obtain a 3D point cloud in the local map that matches the feature points of the current frame;

[0021] The 3D point cloud is subjected to PnP calculation. If the PnP calculation is successful, the result of the PnP calculation is used as the initial pose of the current frame.

[0022] If the PnP calculation fails, the result obtained from the inertial navigation integral calculation is used as the initial pose of the current frame.

[0023] In conjunction with the fifth possible implementation of the first aspect, the drift correction amount includes: a correction offset for deta_R rotation and a correction offset for delta_p translation; if the timestamps of the front-end keyframes and the back-end keyframes are not the same, the front-end tracking module is specifically used for:

[0024] The pose and velocity of the front-end keyframes are optimized according to the following formula;

[0025]

[0026] Where v is velocity, R and p are pose (rotation of pose and translation of position) between two coordinate systems respectively, [u,v] are the pixel coordinates of the current frame, π(·) is the perspective projection function, the superscripts and subscripts w, b and c represent the world coordinate system, the carrier coordinate system and the camera coordinate system respectively, and λ is the inverse depth.

[0027] In conjunction with the sixth possible implementation of the first aspect, the drift correction amount includes: a correction offset for deta_R rotation and a correction offset for delta_p translation. If the timestamps of the front-end keyframes and the back-end keyframes are not the same, the front-end tracking module is specifically used for:

[0028] The pose and velocity of the front-end keyframes are optimized according to the following formula;

[0029]

[0030] Where [u,v] are the pixel coordinates of the current frame, π(·) is the perspective projection function, R and p are the poses, which are the rotation of the pose and the translation of the position, respectively, between the two coordinate systems. The superscripts and subscripts w, b and c represent the world coordinate system, the carrier coordinate system and the camera coordinate system, respectively. λ is the inverse depth, α, β and γ are the inertial navigation pre-integration terms corresponding to position, velocity and attitude, respectively, g is the gravitational acceleration, Δt is the time difference between two frames, and v is the velocity.

[0031] In conjunction with the seventh possible implementation of the first aspect, the front-end tracking module is also used for:

[0032] When adding an anchor point, record the timestamp of the anchor point frame and the anchor point's position in the camera coordinate system. The anchor frame is the image frame to which the anchor point has been added;

[0033] After the loop closure optimization module performs a loop closure update, it uses the recorded timestamps to find the position of the anchor frame in the loop closure keyframe sequence.

[0034] The new pose of the anchor frame after drift elimination is calculated using an interpolation method.

[0035] Based on the following formula, the formula is as follows: Update The This represents the position of the anchor point in the world coordinate system.

[0036] In conjunction with the eighth possible implementation of the first aspect, the front-end tracking module is also used for:

[0037] The new pose after the anchor frame eliminates drift is calculated using the following formula.

[0038]

[0039] Where t is the timestamp of the anchor frame, t1 and t2 are the timestamps of the loopback keyframes, and t1 < t < t2.

[0040] The display device provided in this embodiment, if the timestamp of the front-end keyframe is the same as the timestamp of the back-end keyframe, then uses the optimized pose and velocity of the back-end keyframe as the optimized pose and velocity of the front-end keyframe. Therefore, it can reduce the computational load of the front-end tracking module to a certain extent, achieving the goal of real-time output of the optimized pose of the current frame. Furthermore, after the loop closure optimization module performs a loop closure update, the pose of each anchor point in the anchor frame can be optimized. When the camera returns to the historical area, the virtual object (anchor point) will still be seen in its original position, eliminating its drift in the map and improving the AR / VR user experience.

[0041] A second aspect of this application provides a virtual object pose update method, the method being applicable to a display device, the display device comprising: a front-end tracking module, a back-end optimization module, and a loop closure optimization module.

[0042] The backend optimization module is used to optimize the backend keyframes based on the received IMU data to obtain the optimized pose and velocity of the backend keyframes, wherein the backend keyframes are keyframes of the local map.

[0043] The loop closure optimization module is used to perform SLAM loop closure update on the received camera data to obtain the drift correction amount;

[0044] The front-end tracking module is used to receive IMU data, camera data, optimized pose and velocity of back-end keyframes, and drift correction amount. The camera data includes front-end keyframes.

[0045] If the timestamp of the front-end keyframe is different from the timestamp of the back-end keyframe, then the variables of the front-end keyframe are optimized based on the IMU data, camera data, and drift correction amount.

[0046] If the timestamp of the front-end keyframe is the same as the timestamp of the back-end keyframe, then the optimized pose and velocity of the back-end keyframe are used as the optimized pose and velocity of the front-end keyframe. Based on the camera data, the optimized pose and velocity of the front-end keyframe, the drift correction amount, and other variables of the front-end keyframe are optimized. The other variables are the variables other than pose and velocity.

[0047] The method provided in this embodiment, if the timestamp of the front-end keyframe is the same as the timestamp of the back-end keyframe, then uses the optimized pose and velocity of the back-end keyframe as the optimized pose and velocity of the front-end keyframe. Therefore, it can reduce the computational load of the front-end tracking module to a certain extent, achieving the goal of real-time output of the optimized pose of the current frame. Furthermore, after the loop closure optimization module performs a loop closure update, the pose of each anchor point in the anchor frame can be optimized. When the camera returns to the historical area, the virtual object (anchor point) will still be seen in its original position, eliminating its drift in the map and improving the AR / VR user experience. Attached Figure Description

[0048] To more clearly illustrate the implementation methods in the embodiments of this application or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0049] Figure 1 This illustrates a use case of a display device according to some embodiments;

[0050] Figure 2 A hardware configuration block diagram of a control device 100 according to some embodiments is shown;

[0051] Figure 3 A hardware configuration block diagram of a display device 200 according to some embodiments is shown;

[0052] Figure 4 A software configuration diagram of a display device 200 according to some embodiments is shown;

[0053] Figure 5 A structural block diagram of the controller in a display device provided in a feasible embodiment;

[0054] Figure 6 Here is the workflow diagram for the controller;

[0055] Figure 7 A diagram illustrating the movement of the anchor point during the anchor point positioning process, provided as a feasible embodiment.

[0056] Figure 8 An anchor point pose optimization flowchart provided for a feasible embodiment;

[0057] Figure 9 A schematic diagram of an interpolation-based anchor frame pose correction algorithm provided as a feasible embodiment;

[0058] Figure 10 Schematic diagram for anchor point pose calculation. Detailed Implementation

[0059] To make the objectives and implementation methods of this application clearer, the exemplary implementation methods of this application will be clearly and completely described below with reference to the accompanying drawings of the exemplary embodiments of this application. Obviously, the exemplary embodiments described are only some embodiments of this application, and not all embodiments.

[0060] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0061] The terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar or related objects or entities, and do not necessarily imply a specific order or sequence, unless otherwise specified. It should be understood that such terms are interchangeable where appropriate.

[0062] The terms “comprising” and “having”, and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device that includes a range of components is not necessarily limited to all of the components that are clearly listed, but may include other components that are not clearly listed or that are inherent to such product or device.

[0063] The term "module" refers to any known or subsequently developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0064] Figure 1 This is a schematic diagram illustrating a usage scenario of the display device according to an embodiment. For example... Figure 1 As shown, the display device 200 also communicates with the server 400, and the user can operate the display device 200 through the smart device 300 or the control device 100.

[0065] In some embodiments, the control device 100 may be a remote control. Communication between the remote control and the display device includes at least one of infrared protocol communication, Bluetooth protocol communication, and other short-range communication methods, controlling the display device 200 wirelessly or via a wired connection. Users can control the display device 200 by inputting user commands through at least one method, such as buttons on the remote control, voice input, or control panel input.

[0066] In some embodiments, the smart device 300 may include any one of a mobile terminal, tablet computer, computer, laptop computer, AR / VR device, etc.

[0067] In some embodiments, a smart device 300 may also be used to control the display device 200. For example, an application running on the smart device may be used to control the display device 200.

[0068] In some embodiments, the smart device 300 and the display device may also be used for data communication.

[0069] In some embodiments, the display device 200 can also be controlled in ways other than the control device 100 and the smart device 300. For example, it can be controlled by directly receiving the user's voice commands through a module configured inside the display device 200 for acquiring voice commands, or it can be controlled by receiving the user's voice commands through a voice control device set outside the display device 200.

[0070] In some embodiments, the display device 200 also communicates with the server 400. The display device 200 may communicate via a local area network (LAN), wireless local area network (WLAN), and other networks. The server 400 may provide various content and interactive features to the display device 200. The server 400 may be a cluster or multiple clusters, and may include one or more types of servers.

[0071] In some embodiments, software steps executed by one execution entity can be migrated to another execution entity with which it communicates data, as needed. For example, software steps executed by a server can be migrated to a display device with which it communicates data, and vice versa.

[0072] Figure 2 An exemplary block diagram of the configuration of the control device 100 according to an exemplary embodiment is shown. Figure 2 As shown, the control device 100 includes a controller 110, a communication interface 130, a user input / output interface 140, a memory, and a power supply. The control device 100 can receive user input operation commands and convert the operation commands into commands that the display device 200 can recognize and respond to, thus acting as an intermediary for interaction between the user and the display device 200.

[0073] In some embodiments, the communication interface 130 is used for external communication and includes at least one of a WIFI chip, a Bluetooth module, an NFC module, or an alternative module.

[0074] In some embodiments, the user input / output interface 140 includes at least one of a microphone, touchpad, sensor, button, or alternative module.

[0075] Figure 3 A hardware configuration block diagram of a display device 200 according to an exemplary embodiment is shown.

[0076] In some embodiments, the display device 200 includes at least one of a tuner 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface.

[0077] In some embodiments, the controller includes a central processing unit, a video processor, an audio processor, a graphics processor, RAM, ROM, and a first to an nth interface for input / output.

[0078] In some embodiments, the display 260 includes a display screen component for presenting an image, a driving component for driving image display, a component for receiving image signals output from a controller, and a user control UI interface, etc.

[0079] In some embodiments, the display 260 may be at least one of a liquid crystal display, an OLED display, and a projection display, and may also be a projection device and a projection screen.

[0080] In some embodiments, the tuner 210 receives broadcast display device signals via wired or wireless reception, and demodulates audio and video signals, such as EPG data signals, from multiple wireless or wired broadcast display device signals.

[0081] In some embodiments, the communicator 220 is a component used to communicate with external devices or servers according to various communication protocol types. For example, the communicator may include at least one of a Wi-Fi module, a Bluetooth module, a wired Ethernet module, other network communication protocol chips or near-field communication protocol chips, and an infrared receiver. The display device 200 can establish the transmission and reception of control signals and data signals with the control device 100 or the server 400 through the communicator 220.

[0082] In some embodiments, detector 230 is used to acquire signals from the external environment or to interact with the outside world. For example, detector 230 includes a light receiver, a sensor for acquiring ambient light intensity; or, detector 230 includes an image acquisition device, such as a camera, which can be used to acquire external environmental scenes, user attributes, or user interaction gestures; or, detector 230 includes a sound acquisition device, such as a microphone, for receiving external sounds.

[0083] In some embodiments, the external device interface 240 may include, but is not limited to, one or more interfaces such as: High Definition Multimedia Interface (HDMI), analog or data high-definition component input interface (component), composite video input interface (CVBS), USB input interface (USB), RGB port, etc. It may also be a composite input / output interface formed by multiple interfaces mentioned above.

[0084] In some embodiments, the controller 250 and the tuner 210 may be located in different separate devices, that is, the tuner 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0085] In some embodiments, the controller 250 controls the operation of the display device and responds to user operations via various software control programs stored in memory. The controller 250 controls the overall operation of the display device 200. For example, in response to receiving a user command to select a UI object to display on the display 260, the controller 250 can perform operations related to the object selected by the user command.

[0086] In some embodiments, the object can be any of the optional objects, such as a hyperlink, an icon, or other operable area. Operations related to the selected object include: displaying links to hyperlinked pages, documents, images, etc., or performing operations corresponding to the icon.

[0087] In some embodiments, the controller includes at least one of a central processing unit (CPU), a video processor, an audio processor, a graphics processing unit (GPU), RAM (random access memory), ROM (read-only memory), a first to an nth interface for input / output, a communication bus, etc.

[0088] A CPU (CPU) processor is used to execute operating system and application instructions stored in memory, as well as various interactive instructions received from external input, to execute various applications, data, and content, ultimately displaying and playing various audio and video content. A CPU processor can include multiple processors, such as a main processor and one or more sub-processors.

[0089] In some embodiments, a graphics processor is used to generate at least one of various graphical objects, such as icons, operation menus, and user-input-based graphics. The graphics processor includes an arithmetic logic unit (ALU) that performs calculations based on various user-input interactive commands and displays various objects according to display attributes; it also includes a renderer that renders the various objects obtained from the ALU, the rendered objects being displayed on a monitor.

[0090] In some embodiments, a video processor is configured to receive an external video signal and perform at least one of the following video processing operations according to a standard encoding and decoding protocol of the input signal: decompression, decoding, scaling, noise reduction, frame rate conversion, resolution conversion, image synthesis, etc., to obtain a signal that can be directly displayed or played on a display device 200.

[0091] In some embodiments, the video processor includes at least one of a demultiplexing module, a video decoding module, an image compositing module, a frame rate conversion module, and a display formatting module. The demultiplexing module demultiplexes the input audio and video data streams. The video decoding module processes the demultiplexed video signal, including decoding and scaling. The image compositing module, such as an image synthesizer, overlays and blends a GUI signal generated by a graphics generator based on user input or its own generation with the scaled video image to generate a displayable image signal. The frame rate conversion module converts the input video frame rate. The display formatting module modifies the received frame rate-converted video output signal to conform to a display format, such as outputting RGB data signals.

[0092] In some embodiments, an audio processor is configured to receive external audio signals, perform decompression and decoding according to a standard codec protocol of the input signals, and perform at least one of the following processes: noise reduction, digital-to-analog conversion, and amplification, to obtain a sound signal that can be played in a speaker.

[0093] In some embodiments, the user can input user commands through a graphical user interface (GUI) displayed on the display 260, and the user input interface receives the user input commands through the GUI. Alternatively, the user can input user commands by inputting specific sounds or gestures, and the user input interface receives the user input commands by recognizing the sounds or gestures through sensors.

[0094] In some embodiments, a "user interface" is the medium through which an application or operating system interacts and exchanges information with a user, enabling the conversion between the internal form of information and a form acceptable to the user. A common form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operation displayed graphically. It can be an icon, window, operating area, or other interface element displayed on the screen of an electronic device. The operating area can include at least one of the visual interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and widgets.

[0095] In some embodiments, the user interface 280 is an interface that can be used to receive control input (e.g., physical buttons on the display device body, or others).

[0096] In some embodiments, the display device's system may include a kernel, a command interpreter (shell), a content system, and applications. The kernel, shell, and content system together form the basic operating system structure, allowing users to manage content, run programs, and use the system. Upon power-up, the kernel starts, activates the kernel space, abstracts hardware, initializes hardware parameters, and runs and maintains virtual memory, the scheduler, signals, and inter-thread communication (IPC). After the kernel starts, the shell and user applications are loaded. Applications are compiled into machine code after startup, forming a thread.

[0097] like Figure 4 As shown, a display device's system can include a kernel, a command interpreter (shell), a content system, and applications. The kernel, shell, and content system together form the basic operating system structure, allowing users to manage content, run programs, and use the system. Upon power-up, the kernel starts, activates the kernel space, abstracts hardware, initializes hardware parameters, and runs and maintains virtual memory, the scheduler, signals, and inter-process communication (IPC). After the kernel starts, the shell and user applications are loaded. Applications are compiled into machine code after startup, forming a process.

[0098] like Figure 4 As shown, the display device system is divided into three layers, from top to bottom: the application layer, the middleware layer, and the hardware layer (which can also be referred to as the system operation layer in this embodiment).

[0099] The application layer mainly includes commonly used applications on display devices, as well as application frameworks. Commonly used applications are mainly browser-based applications, such as HTML5 apps, and native apps.

[0100] An application framework is a complete program model that has all the basic functions required by standard application software, such as content access, data exchange, and the user interface for these functions (toolbar, status bar, menu, dialog box).

[0101] Native apps can support online or offline access, push notifications, or access to local resources.

[0102] The middleware layer includes various display device protocols, multimedia protocols, and system components. Middleware can use the basic services (functions) provided by system software to connect different parts of application systems or different applications on the network, achieving resource sharing and function sharing.

[0103] The hardware layer mainly includes the HAL interface, hardware, and drivers. The HAL interface is a unified interface for all display device chips, with the specific logic implemented by each individual chip. The drivers mainly include: audio drivers, display drivers, Bluetooth drivers, camera drivers, Wi-Fi drivers, USB drivers, HDMI drivers, sensor drivers (such as fingerprint sensors, temperature sensors, pressure sensors, etc.), and power drivers.

[0104] In some embodiments, after the display device is started, it can directly enter the display interface of the last selected signal source, or the signal source selection interface. The signal source can be a preset video-on-demand program, or at least one of an HDMI interface, a live broadcast display device interface, etc. After the user selects different signal sources, the display can show content obtained from those different signal sources. This is applicable to [various applications].

[0105] One of the primary applications of AR (Augmented Reality) / VR (Virtual Reality) technology is placing virtual objects in a virtual world. In SLAM (Simultaneous Localization and Mapping), these virtual objects are represented by anchor points. The position of the anchor point represents the position of the virtual object, and its pose represents the pose of the virtual object. Because camera pose drift occurs during long-term SLAM operation, if an anchor point is placed at time t0, its position will be shifted when returning to the same area at time t1. A common approach is to add a loop closure optimization to eliminate camera pose (including position and orientation) drift. However, this loop closure optimization is resource-intensive and difficult to calculate in real-time the amount of pose drift.

[0106] To address the aforementioned technical problems, a first aspect of this application provides a display device, including a display and a controller, wherein the controller includes a front-end tracking module, a back-end optimization module, and a loopback optimization module. The display device will now be described in conjunction with specific accompanying drawings.

[0107] Figure 5 This is a structural block diagram of the controller in a display device provided in a feasible embodiment. Figure 6 This is the workflow diagram for the controller.

[0108] The backend optimization module executes S61 to optimize the backend keyframe based on the received IMU data, thereby obtaining the optimized pose and velocity of the backend keyframe.

[0109] In this embodiment, the backend keyframe is a keyframe in a local map.

[0110] As an alternative, the latest timestamp alignment parameter timeshift is obtained, and all IMU data after timestamp alignment between the previous frame and the current frame is obtained.

[0111] In this embodiment, the measurement may include the object's three-axis attitude angles (or angular rates) and acceleration. The position mentioned in this embodiment may include both position and attitude.

[0112] The keyframes are optimized based on the received IMU data. The optimized pose of the keyframes can be obtained using image frame optimization methods commonly used in this field, which will not be elaborated upon here.

[0113] The loop closure optimization module executes S62 to perform SLAM loop closure update on the received camera data to obtain the drift correction amount;

[0114] The drift correction amount can be obtained by performing SLAM loop closure update on the received camera data. The loop closure update process (also known as the loop closure optimization process) commonly used in this field can be adopted, and the applicant will not impose any further restrictions here.

[0115] The front-end tracking module executes S63, receiving IMU data, camera data, and optimized pose, velocity, and drift correction amounts from back-end keyframes.

[0116] In this embodiment, the camera data includes continuously acquired image frames from the camera, where each image frame corresponds to a timestamp. In this embodiment, the image frames processed by the front-end tracking module are referred to as front-end keyframes.

[0117] If the timestamp of the front-end keyframe is the same as the timestamp of the back-end keyframe, the front-end tracking module executes S64 and calls the optimized pose and velocity of the back-end keyframe as the optimized pose and velocity of the front-end keyframe. Based on the camera data, the optimized pose and velocity of the front-end keyframe, the drift correction amount, and other variables of the front-end keyframe are optimized. The other variables are the variables other than pose and velocity.

[0118] If the timestamp of the front-end keyframe is different from the timestamp of the back-end keyframe, the front-end tracking module executes S65 to optimize the variables of the front-end keyframe based on the IMU data, camera data, and drift correction amount.

[0119] There are multiple ways to calculate the optimization variables of the front-end keyframes;

[0120] In some feasible implementations, the front-end tracking module can use inertial navigation pre-integration terms to perform inertial navigation calculations on the pose of the previous frame to obtain the initial pose of the current frame. In this embodiment, the current frame is the front-end keyframe being processed by the front-end tracking module, and the previous frame is the previous front-end keyframe of the current frame.

[0121] The calculation process of the inertial navigation pre-integration term can be as follows: calculate the inertial navigation pre-integration term of the previous frame and the current frame, where the previous frame is the image frame preceding the current frame. For example, in a feasible embodiment, the inertial navigation pre-integration term can be obtained according to the following formula.

[0122]

[0123] Where R and p are the rotation and translation of the optimized pose between two coordinate systems, respectively. The superscripts and subscripts w, b, and c represent the world coordinate system, the carrier coordinate system, and the camera coordinate system, respectively. λ is the inverse depth, α, β, and γ are the inertial navigation pre-integration terms corresponding to position, velocity, and attitude, respectively. g is the gravity acceleration, Δt is the time difference between two frames, and v is the velocity.

[0124] In this embodiment, the carrier coordinate system refers to a coordinate system with the carrier's centroid as the origin, OX along the vertical axis (i.e., the carrier's forward direction), Z along the carrier's lateral axis, and Y along the carrier's vertical axis. In this embodiment, the camera coordinate system (also called the observation coordinate system) has its origin at the camera's optical center, with the x and y axes parallel to the image's x and y axes, and the z axis being the camera's optical axis, perpendicular to the image plane. The intersection of the optical axis and the image plane is the origin of the image coordinate system, which is a two-dimensional rectangular coordinate system. In this embodiment, the world coordinate system is the system's absolute coordinate system. Before establishing the user coordinate system, the coordinates of all points on the screen are determined by the origin of this coordinate system.

[0125] In some feasible implementations, the front-end tracking module can utilize a 3D point cloud that matches the feature points of the current frame in the local map; perform PnP calculation on the 3D point cloud to obtain the initial pose of the current frame.

[0126] In this embodiment, the 3D point cloud is a dataset of three-dimensional coordinate points arranged in a regular grid. In this embodiment, Perspective-n-Point (PnP) calculation is a method for solving the motion of 3D to 2D point pairs. If the 3D position of a feature point in one of two images is known, then only a minimum of three point pairs are needed to estimate camera motion using PnP. Specifically, in the scheme applied to this embodiment, performing PnP calculation on the 3D point cloud yields the initial pose of the current frame.

[0127] In some feasible implementations, the front-end tracking module can utilize: inertial navigation pre-integration terms to perform inertial navigation calculations on the pose of the previous frame; obtain a 3D point cloud in the local map that matches the feature points of the current frame; perform PnP calculations on the 3D point cloud; if the PnP calculation is successful, use the result of the PnP calculation as the initial pose of the current frame; if the PnP calculation is unsuccessful, use the result of the inertial navigation integration calculation as the initial pose of the current frame.

[0128] In some feasible implementations, the drift correction includes: a correction offset for deta_R rotation and a correction offset for delta_p translation. If the timestamps of the front-end keyframes and the back-end keyframes are not the same, the front-end tracking module is specifically used for: R and p being the pose transformation between two coordinate systems, representing the rotation of the pose and the translation of the position.

[0129]

[0130] Where v is velocity, R and p are pose (rotation of pose and translation of position) between two coordinate systems respectively, [u,v] are the pixel coordinates of the current frame, π(·) is the perspective projection function, the superscripts and subscripts w, b and c represent the world coordinate system, the carrier coordinate system and the camera coordinate system respectively, and λ is the inverse depth.

[0131] During the above optimization process, Optimize while keeping the size as small as possible.

[0132] To improve the accuracy of the optimized pose in the current frame, as a feasible approach, the result calculated by PnP can be further optimized using Formula 2, or the result calculated by the inertial navigation integral can be further optimized using Formula 2.

[0133] In some feasible implementations, the front-end tracking module is specifically used for: R and p being the poses, which are the rotation of the pose and the translation of the position, respectively, between two coordinate systems;

[0134]

[0135]

[0136] Where [u,v] are the pixel coordinates of the current frame, π(·) is the perspective projection function, R and p are the poses, which are the rotation of the pose and the translation of the position, respectively, between the two coordinate systems. The superscripts and subscripts w, b and c represent the world coordinate system, the carrier coordinate system and the camera coordinate system, respectively. λ is the inverse depth, α, β and γ are the inertial navigation pre-integration terms corresponding to position, velocity and attitude, respectively, g is the gravitational acceleration, Δt is the time difference between two frames, and v is the velocity.

[0137] During the above optimization process, and Optimize while keeping the size as small as possible.

[0138] To improve the accuracy of the optimized pose in the current frame, as a feasible approach, the result calculated by PnP can be further optimized using Formula 1 and Formula 2, or the result calculated by the inertial navigation integral can be further optimized using Formula 1 and Formula 2.

[0139] As a feasible implementation method, if the window of the front-end tracking module is not full (i.e., the number of image frames processed in the front-end tracking module has not reached the preset number), then return directly to step 61; if the sliding window is full (i.e., the number of image frames processed in the front-end tracking module is equal to the preset number), then remove the oldest frame first and then return to step 61.

[0140] The display device provided in this embodiment, if the timestamp of the front-end keyframe is the same as the timestamp of the back-end keyframe, then uses the optimized pose and velocity of the back-end keyframe as the optimized pose and velocity of the front-end keyframe. Therefore, it can reduce the computational load of the front-end tracking module to a certain extent, achieving the goal of real-time output of the optimized pose of the current frame.

[0141] Optimizing only the pose of the current frame without optimizing the anchor point poses will not allow the anchor points to return to their original positions completely. Furthermore, in the case of multiple anchor points, due to the dependencies between them, correcting a single anchor point will directly affect the positions of other anchor points. For example... Figure 7 This diagram illustrates the movement of anchor points during anchor point positioning, as provided in a feasible embodiment. In Effect 1, it can be seen that the first anchor point (anchor point a) is placed at the center A. After the pose of the image frame drifts, the first anchor point no longer coincides with the center A. For details, please refer to Effect 2. When the drift of the first anchor point is eliminated, due to the previous interdependence of the anchor points, the second anchor point (anchor point b) shifts from its original position. For details, please refer to Effect 3.

[0142] To address the aforementioned technical problems, this embodiment further optimizes the display device provided in the above embodiments. For details, please refer to... Figure 8 , Figure 8 An anchor point pose optimization flowchart is provided for a feasible embodiment. Based on the display device provided in the above embodiment, the front-end tracking module is further configured to execute S81-S84:

[0143] When adding an anchor point, the S81 records the timestamp of the anchor point frame and the anchor point's position in the camera coordinate system. The anchor frame is the image frame to which the anchor point has been added;

[0144] In this embodiment, the camera coordinate system (also known as the observation coordinate system) has its origin at the optical center of the camera. The x-axis and y-axis are parallel to the X and Y axes of the image, and the z-axis is the camera's optical axis, which is perpendicular to the image plane. The intersection of the optical axis and the image plane is the origin of the image coordinate system, which is a two-dimensional Cartesian coordinate system.

[0145] After the loop closure optimization module performs a loop closure update, S82 uses the recorded timestamps to find the position of the anchor frame in the loop closure keyframe sequence.

[0146] S83 calculates the new pose of the anchor frame after drift elimination using an interpolation method.

[0147] During SLAM loop closure, each keyframe receives a corresponding drift correction, which corrects the overall keyframe position. However, anchor frames are not necessarily keyframes, so they lack a corresponding drift correction and cannot be directly corrected. Interpolation methods can be used to interpolate the corrected poses of the anchor frames based on the poses of the frames before and after them, yielding a more accurate corrected pose.

[0148] The interpolation method will be explained below with reference to specific appendages. Figure 9 This is a schematic diagram of an interpolation-based anchor frame pose correction algorithm provided in a feasible embodiment. It can be seen that the anchor frame is not a keyframe; the new pose after the anchor frame ticket is removed can be calculated using the poses of keyframes t1 and t2.

[0149] The new pose after anchor frame ticket removal is calculated using the poses of keyframes t1 and t2. There are several ways to implement this. In one feasible embodiment, the new pose after the anchor frame eliminates drift can be calculated according to the following formula.

[0150] Where t is the timestamp of the anchor frame, t1 and t2 are the timestamps of the loopback keyframes, and t1 < t < t2.

[0151] S84 is based on the following formula: Update The This represents the position of the anchor point in the world coordinate system.

[0152] The principle of anchor frame update will be explained below. Figure 10 Anchor point pose calculation diagram. In AR / VR interaction, anchor points / virtual objects are sometimes added by clicking the screen. For example... Figure 10 As shown, when a point (u,v) in the image plane π is clicked, the line connecting the optical center O and the image point (u,v) intersects the real-world plane Ω at point O. At that point, the position of the anchor point / virtual object is the intersection point.

[0153] Through this intersecting geometric calculation, the position of the anchor point / virtual object in the camera coordinate system can be obtained. The pose of the anchor frame can be obtained from the SLAM loop closure update module. Combining both methods allows us to calculate the position of the anchor point / virtual object in the world coordinate system:

[0154]

[0155] In the display device provided in this embodiment, if the timestamp of the image frame in the front-end tracking module matches the timestamp of the keyframe in the local map, the pose of the image frame is replaced with the pose of the local map keyframe after drift elimination (i.e., the optimized pose of the keyframe) at the corresponding timestamp. Therefore, the computational load of the front-end tracking module can be reduced to a certain extent, achieving the goal of real-time output of the optimized pose of the current frame. Furthermore, after the loop closure optimization module performs a loop closure update, the pose of each anchor point in the anchor frame can be optimized. When the camera returns to the historical area, the virtual object (anchor point) will still be seen in its original position, eliminating its drift in the map and improving the AR / VR user experience.

[0156] A second aspect of this application provides a virtual object pose update method, the method being applicable to a display device, the display device comprising: a front-end tracking module, a back-end optimization module, and a loop closure optimization module.

[0157] The backend optimization module is used to optimize keyframes based on the received IMU data to obtain the optimized pose and velocity of the backend keyframes.

[0158] The loop closure optimization module is used to perform SLAM loop closure update on the received camera data to obtain the drift correction amount;

[0159] The front-end tracking module is used to receive IMU data, camera data, optimized pose and velocity of back-end keyframes, and drift correction amount. The camera data includes keyframes.

[0160] If the timestamp of the front-end keyframe is different from the timestamp of the back-end keyframe, then the optimization variables of the front-end keyframe are calculated based on the IMU data, camera data and drift correction amount.

[0161] If the timestamp of the front-end keyframe is the same as the timestamp of the back-end keyframe, then the optimized pose and velocity of the back-end keyframe are used as the optimized pose and velocity of the front-end keyframe. Based on the camera data, the optimized pose and velocity of the front-end keyframe, and the drift correction amount, other variables of the front-end keyframe are calculated. The other variables are the variables other than pose and velocity in the optimization variables.

[0162] Optionally, the front-end tracking module is specifically used for:

[0163] Calculate the inertial navigation pre-integration term between the previous frame and the current frame, where the current frame is the front-end key frame being processed by the front-end tracking module, and the previous frame is the front-end key frame preceding the current frame.

[0164] Optionally, the front-end tracking module is specifically used for:

[0165] The inertial navigation pre-integration term is used to perform inertial navigation calculation on the pose of the previous frame to obtain the initial pose of the current frame.

[0166] Optionally, the front-end tracking module is specifically used for:

[0167] Obtain a 3D point cloud in the local map that matches the feature points of the current frame;

[0168] The initial pose of the current frame is obtained by performing PnP calculation on the 3D point cloud.

[0169] Optionally, the front-end tracking module is specifically used for:

[0170] The pose of the previous frame is calculated using the inertial navigation pre-integration term;

[0171] Obtain a 3D point cloud in the local map that matches the feature points of the current frame;

[0172] The 3D point cloud is subjected to PnP calculation. If the PnP calculation is successful, the result of the PnP calculation is used as the initial pose of the current frame.

[0173] If the PnP calculation fails, the result obtained from the inertial navigation integral calculation is used as the initial pose of the current frame.

[0174] Optionally, the drift correction includes: a correction offset for deta_R rotation and a correction offset for delta_p translation; the front-end tracking module is specifically used for:

[0175] The pose and velocity of the front-end keyframes are optimized according to the following formula;

[0176]

[0177] Where v is velocity, R and p are pose (rotation of pose and translation of position) between two coordinate systems respectively, [u,v] are the pixel coordinates of the current frame, π(·) is the perspective projection function, the superscripts and subscripts w, b and c represent the world coordinate system, the carrier coordinate system and the camera coordinate system respectively, and λ is the inverse depth.

[0178] Optionally, the drift correction includes: a correction offset for deta_R rotation and a correction offset for delta_p translation. The front-end tracking module is specifically used for:

[0179] The pose of the current frame is optimized according to the following formula;

[0180]

[0181] Where [u,v] are the pixel coordinates of the current frame, π(·) is the perspective projection function, R and p are the rotation and translation of the optimized pose transformation between two coordinate systems, respectively, and the superscripts w, b and c represent the world coordinate system, the carrier coordinate system and the camera coordinate system, respectively. λ is the inverse depth, α, β and γ are the inertial navigation pre-integration terms corresponding to position, velocity and attitude, respectively, g is the gravitational acceleration, Δt is the time difference between two frames, and v is the velocity.

[0182] Optionally, the front-end tracking module is also used for:

[0183] When adding an anchor point, record the timestamp of the anchor point frame and the anchor point's position in the camera coordinate system. The anchor frame is the image frame to which the anchor point has been added;

[0184] After the loop closure optimization module performs a loop closure update, it uses the recorded timestamps to find the position of the anchor frame in the loop closure keyframe sequence.

[0185] The new pose of the anchor frame after drift elimination is calculated using an interpolation method.

[0186] Based on the following formula, the formula is as follows: Update The This represents the position of the anchor point in the world coordinate system.

[0187] Optionally, the front-end tracking module is also used for:

[0188] The new pose after the anchor frame eliminates drift is calculated using the following formula.

[0189]

[0190] Where t is the timestamp of the anchor frame, t1 and t2 are the timestamps of the loopback keyframes, and t1 < t < t2.

[0191] The method provided in this embodiment, if the timestamp of the front-end keyframe is the same as the timestamp of the back-end keyframe, then uses the optimized pose and velocity of the back-end keyframe as the optimized pose and velocity of the front-end keyframe. Therefore, it can reduce the computational load of the front-end tracking module to a certain extent, achieving the goal of real-time output of the optimized pose of the current frame. Furthermore, after the loop closure optimization module performs a loop closure update, the pose of each anchor point in the anchor frame can be optimized. When the camera returns to the historical area, the virtual object (anchor point) will still be seen in its original position, eliminating its drift in the map and improving the AR / VR user experience.

[0192] The same or similar parts between the various embodiments in this specification can be referred to mutually. In particular, for network devices / nodes or apparatus devices, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the description in the method embodiments.

[0193] In a specific implementation, the present invention also provides a computer storage medium, wherein the computer storage medium may store a program, and when the program is executed, it may include some or all of the steps of the various embodiments of the custom control button method and the startup method provided by the present invention. The storage medium may be a magnetic disk, an optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0194] This application also provides a chip, which is connected to or includes a memory, for reading and executing software programs stored in the memory, and a method provided in this application.

[0195] This application also provides a computer program product, which includes one or more computer program instructions. When a computer loads and executes the computer program instructions, all or part of the processes or functions according to the various embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. When it runs on the computer, it causes the computer to perform the methods provided in the embodiments of this application.

[0196] Those skilled in the art will clearly understand that the techniques in the embodiments of the present invention can be implemented using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or certain parts of the embodiments of the present invention.

[0197] For ease of explanation, the above description has been provided in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Various modifications and variations can be obtained based on the above teachings. The selection and description of the above embodiments are for the purpose of better explaining the principles and practical applications, thereby enabling those skilled in the art to better utilize the embodiments and various different variations of embodiments suitable for specific application considerations.

[0198] It should be understood that in the various embodiments of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of this application.

[0199] Furthermore, the terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Additionally, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

Claims

1. A display device, characterized in that, It includes a front-end tracking module, a back-end optimization module, and a loop optimization module; The backend optimization module is used to optimize the backend keyframes based on the received IMU data to obtain the optimized pose and velocity of the backend keyframes, wherein the backend keyframes are keyframes of the local map. The loop closure optimization module is used to perform SLAM loop closure update on the received camera data to obtain the drift correction amount; The front-end tracking module is used to receive IMU data, camera data, optimized pose and velocity of back-end keyframes, and drift correction amount. The camera data includes front-end keyframes. If the timestamp of the front-end keyframe is different from the timestamp of the back-end keyframe, then the variables of the front-end keyframe are optimized based on the IMU data, camera data, and drift correction amount. If the timestamp of the front-end keyframe is the same as the timestamp of the back-end keyframe, then the optimized pose and velocity of the back-end keyframe are used as the optimized pose and velocity of the front-end keyframe. Based on the camera data, the optimized pose and velocity of the front-end keyframe, the drift correction amount, and other variables of the front-end keyframe are optimized. The other variables are the variables other than pose and velocity.

2. The display device according to claim 1, characterized in that, The front-end tracking module is specifically used for: Calculate the inertial navigation pre-integration terms for the previous frame and the current frame, where the current frame is the latest front-end key frame being processed by the front-end tracking module, and the previous frame is the front-end key frame preceding the current frame.

3. The display device according to claim 2, characterized in that, The front-end tracking module is specifically used for: The inertial navigation pre-integration term is used to perform inertial navigation calculation on the pose of the previous frame to obtain the initial pose of the current frame.

4. The display device according to claim 2, characterized in that, The front-end tracking module is specifically used for: Obtain a 3D point cloud in the local map that matches the feature points of the current frame; The initial pose of the current frame is obtained by performing PnP calculation on the 3D point cloud.

5. The display device according to claim 2, characterized in that, The front-end tracking module is specifically used for: The pose of the previous frame is calculated using the inertial navigation pre-integration term; Obtain a 3D point cloud in the local map that matches the feature points of the current frame; The 3D point cloud is subjected to PnP calculation. If the PnP calculation is successful, the result of the PnP calculation is used as the initial pose of the current frame. If the PnP calculation fails, the result obtained from the inertial navigation calculation is used as the initial pose of the current frame.

6. The display device according to claim 1, characterized in that, The drift correction includes: a correction offset for delta_R rotation and a correction offset for delta_p translation; if the timestamps of the front-end keyframes and the back-end keyframes are not the same, the front-end tracking module is specifically used for: The pose and velocity of the front-end keyframes are optimized according to the following formula; ； in, and The pose represents the rotation of the orientation and the translation of the position between two coordinate systems, respectively, and u is the pixel coordinate of the feature point in the image. It's a perspective projection function, with subscripts and superscripts. , and Let i and j represent the world coordinate system, the carrier coordinate system, and the camera coordinate system, respectively. Let λ be the inverse depth, and let i and j be the i-th and j-th frames, respectively.

7. The display device according to claim 2, characterized in that, The drift correction includes: a correction offset for delta_R rotation and a correction offset for delta_p translation. If the timestamps of the front-end keyframes and the back-end keyframes are not the same, the front-end tracking module is specifically used for: The pose and velocity of the front-end keyframes are optimized according to the following formula; ； ； Where u is the pixel coordinate of the feature point in the image. It is a perspective projection function. and The pose represents the rotation of the orientation and the translation of the position between two coordinate systems, with superscripts and subscripts indicating the transformation. , and Let these represent the world coordinate system, the vehicle coordinate system, and the camera coordinate system, respectively, with λ representing the inverse depth. , and These are the inertial navigation pre-integration terms between two adjacent frames corresponding to position, velocity, and attitude, respectively. Acceleration due to gravity, Let v be the time difference between two frames, v be the velocity, and the subscripts i and j represent the i-th frame and the j-th frame, respectively.

8. The display device according to any one of claims 1-7, characterized in that, The front-end tracking module is also used for: When adding an anchor point, record the timestamp of the anchor point frame and the anchor point's position in the camera coordinate system. The anchor frame is the image frame to which the anchor point has been added; After the loop closure optimization module performs a loop closure update, it uses the recorded timestamps to find the position of the anchor frame in the loop closure keyframe sequence. The new pose of the anchor frame after drift elimination is calculated using an interpolation method. ; Based on the following formula, the formula is as follows: Update The This represents the position of the anchor point in the world coordinate system.

9. The display device according to claim 8, characterized in that, The front-end tracking module is also used for: The new pose after the anchor frame eliminates drift is calculated using the following formula. ; ； in, For the timestamp of the anchor frame, and All are timestamps of loopback keyframes, and , Indicates in Position at any given moment.

10. A method for updating the pose of a virtual object, the method being applicable to a display device, the display device comprising: a front-end tracking module, a back-end optimization module, and a loop closure optimization module, characterized in that: The backend optimization module is used to optimize the backend keyframes based on the received IMU data to obtain the optimized pose and velocity of the backend keyframes, wherein the backend keyframes are keyframes of the local map. The loop closure optimization module is used to perform SLAM loop closure update on the received camera data to obtain the drift correction amount; The front-end tracking module is used to receive IMU data, camera data, optimized pose and velocity of back-end keyframes, and drift correction amount. The camera data includes front-end keyframes. If the timestamp of the front-end keyframe is different from the timestamp of the back-end keyframe, then the variables of the front-end keyframe are optimized based on the IMU data, camera data, and drift correction amount. If the timestamp of the front-end keyframe is the same as the timestamp of the back-end keyframe, then the optimized pose and velocity of the back-end keyframe are used as the optimized pose and velocity of the front-end keyframe. Based on the camera data, the optimized pose and velocity of the front-end keyframe, the drift correction amount, and other variables of the front-end keyframe are optimized. The other variables are the variables other than pose and velocity.

Images