Method and device for determining motion imaging time delay and medium

Abstract

The invention aims to provide a method and device for determining motion imaging time delay and a medium, and the method comprises the steps: obtaining information of a plurality of images obtained by shooting a display screen of an augmented reality device according to a preset shooting frequency through a shooting device in a uniform motion process of the shooting device and the augmented reality device, the shooting device and the augmented reality device correspond to the same movement speed and movement direction, the augmented reality device locates a target object in a current real scene, and corresponding virtual presentation information is displayed in an overlapping mode according to the real-time current position of the target object on a display screen; and determining the motion imaging time delay of the augmented reality equipment according to the corresponding moving distance information in the first image information and the second image information obtained by shooting the virtual presentation information before and after, the corresponding relative distance information of the target object and the virtual presentation information in the second image information, and the display frame duration information of the display screen.

Description

[0001] This application claims priority to CN 202311659367.X (application date 2023-12-05). Technical Field

[0002] The present application relates to the field of communications, and in particular to a technology for determining motion imaging delay. Background Art

[0003] In the field of augmented reality (AR), Motion-To-Photon Latency (MTP) refers to the total time from the start of the user's head movement to the time when the rendered image using the motion parameters becomes visible on the screen. After the user puts on an augmented reality device (for example, smart glasses), a virtual image is displayed in the glasses, which can be superimposed on objects in the real world. When the user moves his head, due to the MTP delay, the changes in the virtual image observed by the human eye on the display screen are slower than the head movement, which then causes dizziness. In order to reduce MTP latency and improve user experience, major manufacturers have tried many breakthroughs: stability of IMU and Camera; advance prediction of posture; optimization of rendering pipeline stability, etc. Summary of the invention

[0004] An object of the present application is to provide a method, device and medium for determining motion imaging delay.

[0005] According to one aspect of the present application, a method for determining motion imaging delay is provided, the method comprising:

[0006] Acquiring a plurality of image information captured by a shooting device according to a preset shooting frequency for a display screen of the augmented reality device during a uniform motion of the shooting device and the augmented reality device, wherein the shooting device and the augmented reality device correspond to the same motion speed and motion direction, the augmented reality device locates a target object in a current real scene, and overlays and displays corresponding virtual presentation information according to the real-time current position of the target object on the display screen;

[0007] The motion imaging delay of the augmented reality device is determined based on the moving distance information corresponding to the first image information and the second image information obtained by shooting the virtual presentation information before and after, the relative distance information corresponding to the target object and the virtual presentation information in the second image information, and the display frame duration information of the display screen, wherein the first image information and the second image information correspond to different display frames of the display screen.

[0008] According to one aspect of the present application, a computer device for determining motion imaging delay is provided, the device comprising:

[0009] Processor; and

[0010] A memory arranged to store computer executable instructions which, when executed, cause the processor to perform the following operations:

[0011] Acquiring a plurality of image information captured by a shooting device according to a preset shooting frequency for a display screen of the augmented reality device during a uniform motion of the shooting device and the augmented reality device, wherein the shooting device and the augmented reality device correspond to the same motion speed and motion direction, the augmented reality device locates a target object in a current real scene, and overlays and displays corresponding virtual presentation information according to the real-time current position of the target object on the display screen;

[0012] The motion imaging delay of the augmented reality device is determined based on the moving distance information corresponding to the first image information and the second image information obtained by shooting the virtual presentation information before and after, the relative distance information corresponding to the target object and the virtual presentation information in the second image information, and the display frame duration information of the display screen, wherein the first image information and the second image information correspond to different display frames of the display screen.

[0013] According to one aspect of the present application, a computer readable medium storing instructions is provided, wherein when the instructions are executed, the system performs the following operations:

[0014] Acquiring a plurality of image information captured by a shooting device according to a preset shooting frequency for a display screen of the augmented reality device during a uniform motion of the shooting device and the augmented reality device, wherein the shooting device and the augmented reality device correspond to the same motion speed and motion direction, the augmented reality device locates a target object in a current real scene, and overlays and displays corresponding virtual presentation information according to the real-time current position of the target object on the display screen;

[0015] The motion imaging delay of the augmented reality device is determined based on the moving distance information corresponding to the first image information and the second image information obtained by shooting the virtual presentation information before and after, the relative distance information corresponding to the target object and the virtual presentation information in the second image information, and the display frame duration information of the display screen, wherein the first image information and the second image information correspond to different display frames of the display screen.

[0016] According to one aspect of the present application, a user equipment for determining motion imaging delay is provided, the device comprising:

[0017] A module for obtaining information of a plurality of images captured by a shooting device according to a preset shooting frequency during a uniform motion of the shooting device and the augmented reality device, wherein the shooting device and the augmented reality device correspond to the same motion speed and motion direction, and the augmented reality device locates a target object in a current real scene, and overlays and displays corresponding virtual presentation information according to the real-time current position of the target object on the display screen;

[0018] Module one and module two are used to determine the motion imaging delay of the augmented reality device according to the corresponding moving distance information in the first image information and the second image information obtained by shooting the virtual presentation information before and after, the relative distance information between the target object and the virtual presentation information in the second image information, and the display frame duration information of the display screen, wherein the first image information and the second image information correspond to different display frames of the display screen.

[0019] Compared with the prior art, the present application obtains multiple image information obtained by a shooting device according to a preset shooting frequency for the display screen of the augmented reality device during the uniform motion of the shooting device and the augmented reality device, wherein the shooting device and the augmented reality device correspond to the same motion speed and motion direction, the augmented reality device locates the target object in the current real scene, and displays the corresponding virtual presentation information according to the real-time current position of the target object on the display screen; according to the corresponding moving distance information in the first image information and the second image information obtained by shooting the virtual presentation information before and after, and the corresponding relative distance information between the target object and the virtual presentation information in the second image information, and the display frame duration information of the display screen, the motion imaging delay of the augmented reality device is determined, wherein the first image information and the second image information correspond to different display frames of the display screen. The present application provides a method for measuring the MTP delay during uniform motion, and obtains the MTP delay through the mapping relationship between space and time, which can improve the operability of the MTP measurement, reduce the cost of the MTP measurement, and ensure the effectiveness of the MTP delay optimization. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0021] Figure 1 A flow chart of a method for determining motion imaging delay according to an embodiment of the present application is shown;

[0022] Figure 2A structural diagram of a user equipment for determining motion imaging delay according to an embodiment of the present application is shown;

[0023] Figure 3 A schematic diagram for determining motion imaging delay according to an embodiment of the present application is shown;

[0024] Figure 4 A schematic diagram for determining motion imaging delay according to an embodiment of the present application is shown;

[0025] Figure 5 A schematic diagram for determining motion imaging delay according to an embodiment of the present application is shown;

[0026] Figure 6 An exemplary system is shown that can be used to implement the various embodiments described in this application.

[0027] The same or similar reference numerals in the drawings represent the same or similar components. DETAILED DESCRIPTION

[0028] The present application is described in further detail below in conjunction with the accompanying drawings.

[0029] In a typical configuration of the present application, the terminal, the device of the service network and the trusted party all include one or more processors (eg, a central processing unit (CPU)), an input / output interface, a network interface and a memory.

[0030] Memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash memory. Memory is an example of a computer-readable medium.

[0031] Computer readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer readable instructions, data structures, modules of programs or other data. Examples of computer storage media include, but are not limited to, Phase-Change Memory (PCM), Programmable Random Access Memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of random access memory (RAM), Read-Only Memory (ROM), Electrically-Erasable Programmable Read-Only Memory (EEPROM), Flash memory or other memory technology, Compact Disc Read-Only Memory (CD-ROM), Digital Versatile Disc (DVD) or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0032] The devices referred to in this application include but are not limited to user devices, network devices, or devices formed by integrating user devices and network devices through a network. The user devices include but are not limited to any mobile electronic products that can interact with users (for example, interact with users through a touchpad), such as smart phones, tablet computers, smart glasses, etc. The mobile electronic products can use any operating system, such as Android operating system, iOS operating system, etc. Among them, the network device includes an electronic device that can automatically perform numerical calculations and information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable logic devices (Programmable Logic Device, PLD), field programmable gate arrays (Field Programmable Gate Array, FPGA), digital signal processors (Digital Signal Processor, DSP), embedded devices, etc. The network devices include but are not limited to computers, network hosts, single network servers, multiple network server sets or multiple servers. The cloud is composed of a large number of computers or network servers based on cloud computing (Cloud Computing), wherein cloud computing is a type of distributed computing, a virtual supercomputer composed of a group of loosely coupled computer sets. The network includes but is not limited to the Internet, a wide area network, a metropolitan area network, a local area network, a VPN network, a wireless self-organizing network (Ad Hoc network), etc. Preferably, the device may also be a program running on the user device, the network device, or a device formed by integrating the user device and the network device, the network device, the touch terminal, or the network device and the touch terminal through a network.

[0033] Of course, those skilled in the art should understand that the above-mentioned devices are only examples, and other existing or future devices that are applicable to the present application should also be included in the scope of protection of the present application and are included here by reference.

[0034] In the description of the present application, “plurality” means two or more, unless otherwise clearly and specifically defined.

[0035] Figure 1A flow chart of a method for determining motion imaging delay according to an embodiment of the present application is shown, and the method includes step S11 and step S12. In step S11, the user device obtains a plurality of image information obtained by the shooting device according to a preset shooting frequency for the display screen of the augmented reality device during the uniform motion of the shooting device and the augmented reality device, wherein the shooting device and the augmented reality device correspond to the same motion speed and motion direction, and the augmented reality device locates the target object in the current real scene, and displays the corresponding virtual presentation information according to the real-time current position of the target object on the display screen. In step S12, the user device determines the motion imaging delay of the augmented reality device according to the corresponding moving distance information in the first image information and the second image information obtained by shooting the virtual presentation information before and after, and the corresponding relative distance information of the target object and the virtual presentation information in the second image information, and the display frame duration information of the display screen, wherein the first image information and the second image information correspond to different display frames of the display screen.

[0036] In step S11, the user device obtains multiple image information obtained by the shooting device according to a preset shooting frequency for the display screen of the augmented reality device during the uniform motion of the shooting device and the augmented reality device, wherein the shooting device and the augmented reality device correspond to the same motion speed and motion direction, and the augmented reality device locates the target object in the current real scene, and displays the corresponding virtual presentation information superimposed on the real-time current position of the target object on the display screen. In some embodiments, the shooting device can be any device with a camera, preferably, the shooting device can be an industrial camera or a high-speed camera, and the augmented reality device can be any AR (Augmented Reality) device with a camera and a display screen, such as monocular AR smart glasses or binocular AR smart glasses. In some embodiments, the shooting device and the augmented reality device are both in uniform motion, and the two have the same motion speed and motion direction, that is, the two remain relatively still during the motion. The shooting device shoots the display screen of the augmented reality device at a preset shooting frequency during the motion, and can capture image information corresponding to multiple different moments, wherein the image information includes virtual presentation information (i.e., virtual scene content) displayed on the display screen of the augmented reality device and the target object in the current real scene. In some embodiments, the image information displayed on the display screen of the augmented reality device includes both the real scene content in front of it acquired by the camera of the augmented reality device and the virtual presentation information (i.e., virtual scene content) displayed in the optical machine of the augmented reality device. In this case, the shooting device can obtain image information including virtual presentation information and the target object in the current real scene by shooting the display screen of the augmented reality device. In other embodiments, the image information displayed on the display screen of the augmented reality device only includes the virtual presentation information displayed in the optical machine of the augmented reality device. In this case, when shooting the display screen of the augmented reality device, the shooting device obtains the target object in the current real scene through the display screen, thereby obtaining image information including virtual presentation information and the target object in the current real scene. In some embodiments, such as Figure 3As shown, the virtual presentation information (e.g., gray box) displayed on the display screen of the augmented reality device has the ability to locate and track the target object (e.g., black box diagram) in the real scene, wherein the positioning and tracking method can be a SLAM-related algorithm, a 2D recognition and tracking algorithm, a 3D recognition and tracking algorithm, etc., which is only an example and is not limited here. The target object can be any object in the real scene, as long as it has visually recognizable characteristics. For example, the target object can include a variety of geometric shapes, a plurality of different grayscales, a plurality of different colors, patterns, textures, etc. The augmented reality device will track and locate a target object in the current real scene in real time, and according to the real-time current position corresponding to the target object in the current frame of the display screen, the corresponding virtual presentation information is superimposed and displayed in the current frame, for example, superimposed and displayed at the real-time current position, or superimposed and displayed at a position with a preset direction and a preset distance from the real-time current position.

[0037] In step S12, the user device determines the motion imaging delay of the augmented reality device according to the corresponding moving distance information in the first image information and the second image information obtained by shooting before and after the virtual presentation information, the corresponding relative distance information between the target object and the virtual presentation information in the second image information, and the display frame duration information of the display screen, wherein the first image information and the second image information correspond to different display frames of the display screen. In some embodiments, the first image information and the second image information obtained by shooting before and after are determined from a plurality of image information obtained by shooting, wherein the first image information and the second image information correspond to different display frames of the display screen of the augmented reality device, representing that the first image information and the second image information contain different display frames of the display screen of the augmented reality device, that is, the first image information and the second image information contain image information displayed on the display screen of the augmented reality device at different times. In some embodiments, according to the mapping relationship between space and time, the relative distance information corresponding to the target object and the virtual presentation information in the second image information can be divided by the moving distance information corresponding to the virtual presentation information in the first image information and the second image information to obtain the corresponding ratio, and the ratio can be multiplied by the display frame duration of the display screen of the augmented reality device to obtain the motion imaging delay (MTP) of the augmented reality device, wherein the display frame duration refers to the inter-frame interval corresponding to the display screen, which can be determined according to the refresh frequency of the display screen, for example, the refresh frequency is usually 60HZ, 90HZ, 120HZ, and the corresponding display frame durations are 16.667ms, 11.111ms, and 8.333ms, respectively. The present application provides a method for measuring the MTP delay during uniform motion, and the MTP delay is obtained by the mapping relationship between space and time, which can improve the operability of the MTP measurement, reduce the cost of the MTP measurement, and ensure the effectiveness of the MTP delay optimization.

[0038] In some embodiments, the shooting frequency is greater than the refresh frequency corresponding to the display screen. In some embodiments, the shooting frequency of the shooting device needs to be greater than the refresh frequency of the display screen of the augmented reality device. For example, the shooting frequency is 1000HZ, that is, one image is shot every 1ms. In some embodiments, it is necessary to ensure that the refresh frequency of the display screen of the augmented reality device remains stable, that is, to ensure that the interval between frames of the display screen is the same, so as to reduce measurement errors and ensure accuracy.

[0039] In some embodiments, the shooting device and the augmented reality device are fixed on a robotic arm, and the shooting device and the augmented reality device are synchronously moved at a uniform speed by driving the robotic arm to move at a uniform speed. In some embodiments, the robotic arm can move at a uniform speed in a certain direction, for example, left-right uniform speed movement or up-down uniform speed movement. In some embodiments, the shooting device (such as an industrial camera, a high-speed camera, etc.) and the augmented reality device are fixed on a robotic arm, and the shooting device and the augmented reality device can synchronously move at the same uniform speed by driving the robotic arm to move at a uniform speed.

[0040] In some embodiments, the motion direction corresponding to the uniform motion process only involves a two-dimensional plane. In some embodiments, the shooting device and the augmented reality device perform uniform motion in the X-axis direction (horizontal direction) and the Y-axis direction (vertical direction) of the coordinate system (two-dimensional plane), and the Z-axis direction remains absolutely stationary, so that the rotation and translation-related posture motions in the three-dimensional space can be simplified to translation motions in the two-dimensional plane space.

[0041] In some embodiments, the target object includes any of the following: a QR code; a barcode; a visual reference system code (AprilTag). In some embodiments, the target object may be an object including multiple geometric shapes, multiple different grayscales, multiple different colors, patterns, or textures, and may also be any object among a QR code, a barcode, an AprilTag, and the like.

[0042] In some embodiments, the method further includes: the user device determines the first image information and the second image information from the multiple image information. In some embodiments, it is necessary to determine the first image information and the second image information obtained by the previous and next shots from the multiple image information obtained by shooting, and it is only necessary to satisfy that the first image information and the second image information correspond to different display frames of the display screen. For example, two captured images can be randomly determined from the multiple image information as the first image information and the second image information. For another example, two consecutive captured images can be randomly determined from the multiple image information as the first image information and the second image information.

[0043] In some embodiments, the method further includes: obtaining frame sequence number information of a display frame of the display screen corresponding to each image information in the multiple image information; wherein, determining the first image information and the second image information in the multiple image information includes: determining the first image information and the second image information in the multiple image information according to the frame sequence number information. In some embodiments, it is necessary to first obtain frame sequence number information (for example, the Nth frame) of a display frame of the display screen corresponding to each image information in the multiple image information, and then determine the first image information and the second image information in the multiple image information according to the frame sequence number information corresponding to each image information, as long as the two correspond to different frame sequence number information.

[0044] In some embodiments, the display screen displays the frame number information of the current display frame in real time. In some embodiments, the augmented reality device will display the frame number information of the current display frame in real time on the display screen, so that each image information captured by the shooting device will also contain the frame number information. In some embodiments, the frame number information corresponding to the image information can be automatically obtained by performing image content recognition on each image information, and in other embodiments, the frame number information corresponding to the image information can be obtained manually.

[0045] In some embodiments, the step S12 includes: the user device calculates the product of the ratio information of the relative distance information to the moving distance information and the frame duration information corresponding to the display frequency to obtain the motion imaging delay of the augmented reality device. In some embodiments, the motion imaging delay (MTP) of the augmented reality device can be obtained by calculating the product of the ratio of the relative distance information corresponding to the target object and the virtual presentation information in the second image information to the moving distance information corresponding to the virtual presentation information in the first image information and the second image information and the display frame duration of the display screen of the augmented reality device according to the mapping relationship between space and time.

[0046] In some embodiments, the method further includes: the user device determines the moving distance information and the relative distance information. In some embodiments, the user device needs to determine the relative distance information corresponding to the target object and the virtual presentation information in the second image information from the first image information and the second image information, and the moving distance information corresponding to the virtual presentation information in the first image information and the second image information, which can be automatically determined by visually analyzing the first image information and the second image information, or can be determined by physically measuring the image information. For example, the distance information between the image position information corresponding to a certain position (such as the upper left corner, the center point, the left boundary, etc.) of the virtual presentation information in the first image information and the image position information corresponding to the certain position of the virtual presentation information in the second image information is determined by visual analysis / physical measurement, and the distance information is used as the moving distance information. For example, the distance information between the image position information corresponding to a certain position (such as the upper left corner, the center point, the left boundary, etc.) of the virtual presentation information in the second image information and the image position information corresponding to another position of the target object in the second image information is determined by visual analysis / physical measurement, and the distance information is used as the relative distance information.

[0047] In some embodiments, the moving distance information and the relative distance information belong to pixel distance; wherein, the determining the moving distance information and the relative distance information includes: determining the moving distance information and the relative distance information by visually analyzing the target object and the virtual presentation information in the first image information and the second image information. In some embodiments, the moving distance information and the relative distance information may be pixel distance (number of pixels), then the moving distance information may be automatically determined by visually analyzing the virtual presentation information in the first image information and the second image information (for example, using an opencv (cross-platform computer vision library) algorithm), and the relative distance information may be automatically determined by visually analyzing the target object and the virtual presentation information in the second image information (for example, using an opencv (cross-platform computer vision library) algorithm).

[0048] In some embodiments, the moving distance information and the relative distance information belong to physical distance; wherein, the determining the moving distance information and the relative distance information includes: determining the moving distance information and the relative distance information according to the measurement results obtained by measuring the target object and the virtual presentation information in the first image information and the second image information through a measuring tool. In some embodiments, the moving distance information and the relative distance information can be actual physical distances, and the measurement results obtained by measuring the first image information and the virtual presentation information in the second image information through a measuring tool (e.g., a vernier caliper) can be used as the moving distance information, and the measurement results obtained by measuring the target object and the virtual presentation information in the second image information through a measuring tool (e.g., a vernier caliper) can be used as the relative distance information.

[0049] In some embodiments, the method also includes: the user device determines the theoretical presentation position information corresponding to the virtual presentation information in the second image information based on the first position information corresponding to the target object in the second image information and the relative position relationship between the target object and the virtual presentation information in the display screen when the shooting device and the augmented reality device are in a static state before the uniform motion process; and determines the relative distance information based on the second position information corresponding to the virtual presentation information in the second image information and the theoretical presentation position information. In some embodiments, the relative distance information cannot be directly obtained through visual analysis or measurement with a measuring tool. Instead, only the first position information corresponding to the target object in the second image information and the second position information corresponding to the virtual presentation information in the second image information can be directly obtained. Then, based on the first position information and the relative position relationship (relative direction and relative distance) between the target object and the virtual presentation information in the display screen of the augmented reality device in a stationary state before the shooting device and the augmented reality device are in uniform motion, the theoretical presentation position information corresponding to the virtual presentation information in the second image information needs to be calculated (i.e., assuming the ideal case where the MTP has a delay of 0, the presentation position where the virtual presentation information should theoretically be located in the second image information). Then, the theoretical presentation position information is compared with the second position information corresponding to the virtual presentation information in the second image information to determine the relative distance information.

[0050] In some embodiments, when the shooting device and the augmented reality device are in a static state before the uniform motion process, the first display area of ​​the virtual presentation information in the display screen completely overlaps with the second display area of ​​the target object in the display screen; wherein the method further comprises: determining the relative distance information according to the first position information corresponding to the target object in the second image information and the second position information corresponding to the virtual presentation information in the second image information. In some embodiments, if the shooting device and the augmented reality device are in a static state before the uniform motion process, the first display area of ​​the virtual presentation information in the display screen of the augmented reality device completely overlaps with the second display area of ​​the target object in the display screen, it can be considered that the theoretical presentation position information corresponding to the virtual presentation information in the second image information is always the same as the first position information corresponding to the target object in the second image information, that is, the distance between the theoretical presentation position information and the second position information corresponding to the virtual presentation information in the second image information is always equal to the distance between the first position information corresponding to the target object in the second image information and the second position information, that is, the relative distance information can be directly obtained by visual analysis or measurement with a measuring tool. In some embodiments, the image information displayed on the display screen of the augmented reality device includes both the target object acquired by the camera of the augmented reality device and the virtual presentation information displayed in the optical machine of the augmented reality device. In this case, the second display area of ​​the target object in the display screen refers to the actual display area of ​​the target object on the display screen. In other embodiments, the image information displayed on the display screen of the augmented reality device only includes the virtual presentation information displayed in the optical machine of the augmented reality device. In this case, the second display area of ​​the target object in the display screen refers to the position information corresponding to the target object in the display screen through the display screen. As an example, Figure 5As shown, the high-speed camera (shooting device) and the AR glasses (augmented reality device) move to the left at a uniform speed in the horizontal direction. In the image frames taken by the high-speed camera, it is found that the target object (black box) will evenly translate to the right in each frame, while the virtual presentation information (gray box) will translate to the right at a fixed number of frame intervals. The first image information located in the upper part of the figure is the 666th frame taken by the high-speed camera, which captures the 345th frame of the display screen of the augmented reality device. The second image information located in the lower part of the figure is the 683rd frame taken by the high-speed camera, which captures the 346th frame of the display screen of the augmented reality device. The shorter horizontal line on the left side of the figure is the moving distance information corresponding to the virtual presentation information in the first image information and the second image information, and the longer horizontal line on the right side of the figure is the relative distance information corresponding to the target object and the virtual presentation information in the second image information (that is, the relative distance between the theoretical presentation position information corresponding to the virtual presentation information in the second image information and the second position information corresponding to the virtual presentation information in the second image information).

[0051] In some embodiments, the virtual presentation information includes a display frame of a predetermined shape, and when the shooting device and the augmented reality device are in a static state before the uniform motion process, the display frame on the display screen completely frames the target object. In some embodiments, the virtual presentation information can be a display frame of a predetermined shape, such as a rectangular frame, a square frame, a triangular frame, a circular frame, etc. Figure 4 As shown, if the shooting device and the augmented reality device are in a static state before being in a uniform motion process, the display frame (for example, a gray frame) in the display screen of the augmented reality device just completely frames the target object (for example, a black box image), then in each image information obtained by shooting, the theoretical presentation position information corresponding to the display frame in the image information should always be kept consistent with the current position information corresponding to the target object in the image information.

[0052] In some embodiments, the step S12 includes: the user device determines the motion imaging delay of the second image information relative to the first image information according to the corresponding moving distance information in the first image information and the second image information obtained by shooting the virtual presentation information before and after, and the relative distance information corresponding to the target object and the virtual presentation information in the second image information, wherein the first image information and the second image information correspond to different display frames of the display screen; and determines the motion imaging delay of the augmented reality device according to the motion imaging delays respectively corresponding to the multiple groups of image information in the multiple image information. In some embodiments, multiple groups of image information can be determined from the multiple image information, each group of image information includes two image information obtained by shooting successively, and the two image information respectively correspond to different display frames of the display screen of the augmented reality device. For each group of image information, the MTP corresponding to the group of image information is determined according to the two image information in the group of image information, and then the MTP of the augmented reality device is determined according to the MTPs respectively corresponding to the multiple groups of image information, for example, multiple MTPs are averaged, or a number of values ​​that may have errors are first removed from the multiple MTPs, and then the remaining MTPs are averaged, etc.

[0053] Figure 2 A user device structure diagram for determining motion imaging delay according to an embodiment of the present application is shown, and the device includes a first module 11 and a second module 12. The first module 11 is used to obtain a plurality of image information obtained by a shooting device according to a preset shooting frequency for the display screen of the augmented reality device during the uniform motion of the shooting device and the augmented reality device, wherein the shooting device and the augmented reality device correspond to the same motion speed and motion direction, and the augmented reality device locates the target object in the current real scene, and displays the corresponding virtual presentation information superimposed on the real-time current position of the target object on the display screen. The first and second modules 12 are used to determine the motion imaging delay of the augmented reality device according to the corresponding moving distance information in the first image information and the second image information obtained by shooting the virtual presentation information before and after, and the corresponding relative distance information of the target object and the virtual presentation information in the second image information, and the display frame duration information of the display screen, wherein the first image information and the second image information correspond to different display frames of the display screen.

[0054] A module 11 is used to obtain multiple image information obtained by the shooting device according to a preset shooting frequency for the display screen of the augmented reality device during the uniform motion of the shooting device and the augmented reality device, wherein the shooting device and the augmented reality device correspond to the same motion speed and motion direction, and the augmented reality device locates the target object in the current real scene, and displays the corresponding virtual presentation information superimposed on the real-time current position of the target object on the display screen. In some embodiments, the shooting device can be any device with a camera, preferably, the shooting device can be an industrial camera or a high-speed camera, and the augmented reality device can be any AR (Augmented Reality) device with a camera and a display screen, such as monocular AR smart glasses or binocular AR smart glasses. In some embodiments, the shooting device and the augmented reality device are both in uniform motion, and the two have the same motion speed and motion direction, that is, the two remain relatively still during the motion. The shooting device shoots the display screen of the augmented reality device at a preset shooting frequency during the motion, and can capture image information corresponding to multiple different moments, wherein the image information includes virtual presentation information (i.e., virtual scene content) displayed on the display screen of the augmented reality device and the target object in the current real scene. In some embodiments, the image information displayed on the display screen of the augmented reality device includes both the real scene content in front of it acquired by the camera of the augmented reality device and the virtual presentation information (i.e., virtual scene content) displayed in the optical machine of the augmented reality device. In this case, the shooting device can obtain image information including virtual presentation information and the target object in the current real scene by shooting the display screen of the augmented reality device. In other embodiments, the image information displayed on the display screen of the augmented reality device only includes the virtual presentation information displayed in the optical machine of the augmented reality device. In this case, when shooting the display screen of the augmented reality device, the shooting device obtains the target object in the current real scene through the display screen, thereby obtaining image information including virtual presentation information and the target object in the current real scene. In some embodiments, such as Figure 3As shown, the virtual presentation information (e.g., gray box) displayed on the display screen of the augmented reality device has the ability to locate and track the target object (e.g., black box diagram) in the real scene, wherein the positioning and tracking method can be a SLAM-related algorithm, a 2D recognition and tracking algorithm, a 3D recognition and tracking algorithm, etc., which is only an example and is not limited here. The target object can be any object in the real scene, as long as it has visually recognizable characteristics. For example, the target object can include a variety of geometric shapes, a plurality of different grayscales, a plurality of different colors, patterns, textures, etc. The augmented reality device will track and locate a target object in the current real scene in real time, and according to the real-time current position corresponding to the target object in the current frame of the display screen, the corresponding virtual presentation information is superimposed and displayed in the current frame, for example, superimposed and displayed at the real-time current position, or superimposed and displayed at a position with a preset direction and a preset distance from the real-time current position.

[0055] The first and second modules 12 are used to determine the motion imaging delay of the augmented reality device according to the corresponding moving distance information in the first image information and the second image information obtained by shooting the virtual presentation information before and after, as well as the corresponding relative distance information between the target object and the virtual presentation information in the second image information, and the display frame duration information of the display screen, wherein the first image information and the second image information correspond to different display frames of the display screen. In some embodiments, the first image information and the second image information obtained by shooting before and after are determined from the multiple image information obtained by shooting, wherein the first image information and the second image information correspond to different display frames of the display screen of the augmented reality device, representing that the first image information and the second image information contain different display frames of the display screen of the augmented reality device, that is, the first image information and the second image information contain image information displayed on the display screen of the augmented reality device at different times. In some embodiments, according to the mapping relationship between space and time, the relative distance information corresponding to the target object and the virtual presentation information in the second image information can be divided by the moving distance information corresponding to the virtual presentation information in the first image information and the second image information to obtain the corresponding ratio, and the ratio can be multiplied by the display frame duration of the display screen of the augmented reality device to obtain the motion imaging delay (MTP) of the augmented reality device, wherein the display frame duration refers to the inter-frame interval corresponding to the display screen, which can be determined according to the refresh frequency of the display screen, for example, the refresh frequency is usually 60HZ, 90HZ, 120HZ, and the corresponding display frame durations are 16.667ms, 11.111ms, and 8.333ms, respectively. The present application provides a method for measuring the MTP delay during uniform motion, and the MTP delay is obtained by the mapping relationship between space and time, which can improve the operability of the MTP measurement, reduce the cost of the MTP measurement, and ensure the effectiveness of the MTP delay optimization.

[0056] In some embodiments, the shooting frequency is greater than the refresh frequency corresponding to the display screen. Figure 1 The embodiments shown are the same or similar, so they are not described again and are incorporated herein by reference.

[0057] In some embodiments, the photographing device and the augmented reality device are fixed on a mechanical arm, and the photographing device and the augmented reality device are synchronously moved at a uniform speed by driving the mechanical arm to move at a uniform speed. Figure 1 The embodiments shown are the same or similar, so they are not described again and are incorporated herein by reference.

[0058] In some embodiments, the motion direction corresponding to the uniform motion process only involves a two-dimensional plane. Figure 1 The embodiments shown are the same or similar, so they are not described again and are incorporated herein by reference.

[0059] In some embodiments, the target object includes any one of the following: a QR code; a barcode; a visual reference system code (AprilTag).

[0060] In some embodiments, the device is further configured to: determine the first image information and the second image information from the plurality of image information. Figure 1 The embodiments shown are the same or similar, so they are not described again and are incorporated herein by reference.

[0061] In some embodiments, the device is further used to: obtain frame sequence number information of a display frame of the display screen corresponding to each image information in the multiple image information; wherein, determining the first image information and the second image information in the multiple image information includes: determining the first image information and the second image information in the multiple image information according to the frame sequence number information. Here, the related operations are the same as Figure 1 The embodiments shown are the same or similar, so they are not described again and are incorporated herein by reference.

[0062] In some embodiments, the display screen displays the frame number information of the current display frame in real time. Figure 1 The embodiments shown are the same or similar, so they are not described again and are incorporated herein by reference.

[0063] In some embodiments, the first and second modules 12 are used to calculate the product of the ratio information of the relative distance information to the moving distance information and the frame duration information corresponding to the display frequency to obtain the motion imaging delay of the augmented reality device. Figure 1The embodiments shown are the same or similar, so they are not described again and are incorporated herein by reference.

[0064] In some embodiments, the device is further used to: determine the moving distance information and the relative distance information. Figure 1 The embodiments shown are the same or similar, so they are not described again and are incorporated herein by reference.

[0065] In some embodiments, the moving distance information and the relative distance information belong to pixel distance; wherein, determining the moving distance information and the relative distance information comprises: determining the moving distance information and the relative distance information by visually analyzing the target object and the virtual presentation information in the first image information and the second image information. Figure 1 The embodiments shown are the same or similar, so they are not described again and are incorporated herein by reference.

[0066] In some embodiments, the moving distance information and the relative distance information are physical distances; wherein, determining the moving distance information and the relative distance information comprises: determining the moving distance information and the relative distance information according to a measurement result obtained by measuring the target object and the virtual presentation information in the first image information and the second image information through a measuring tool. Figure 1 The embodiments shown are the same or similar, so they are not described again and are incorporated herein by reference.

[0067] In some embodiments, the device is further used to: determine the theoretical presentation position information corresponding to the virtual presentation information in the second image information according to the first position information corresponding to the target object in the second image information, and the relative position relationship between the target object and the virtual presentation information in the display screen when the shooting device and the augmented reality device are in a static state before the uniform motion process; determine the relative distance information according to the second position information corresponding to the virtual presentation information in the second image information and the theoretical presentation position information. Here, the related operations are as follows Figure 1 The embodiments shown are the same or similar, so they are not described again and are incorporated herein by reference.

[0068] In some embodiments, when the shooting device and the augmented reality device are in a static state before the uniform motion process, the first display area of ​​the virtual presentation information in the display screen completely overlaps with the second display area of ​​the target object in the display screen; wherein the method further comprises: determining the relative distance information according to the first position information corresponding to the target object in the second image information and the second position information corresponding to the virtual presentation information in the second image information. Here, the related operations are the same as Figure 1 The embodiments shown are the same or similar, so they are not described again and are incorporated herein by reference.

[0069] In some embodiments, the virtual presentation information includes a display frame of a predetermined shape, and when the shooting device and the augmented reality device are in a static state before the uniform motion process, the display frame on the display screen just completely frames the target object. Figure 1 The embodiments shown are the same or similar, so they are not described again and are incorporated herein by reference.

[0070] In some embodiments, the one-two module 12 is used to: determine the motion imaging delay of the second image information relative to the first image information according to the moving distance information corresponding to the first image information and the second image information obtained by shooting the virtual presentation information before and after, and the relative distance information corresponding to the target object and the virtual presentation information in the second image information, wherein the first image information and the second image information correspond to different display frames of the display screen; determine the motion imaging delay of the augmented reality device according to the motion imaging delays corresponding to the multiple groups of image information in the multiple image information. Here, the related operations are as follows Figure 1 The embodiments shown are the same or similar, so they are not described again and are incorporated herein by reference.

[0071] In addition to the methods and devices described in the above embodiments, the present application also provides a computer-readable storage medium, which stores computer code. When the computer code is executed, the method described in any of the preceding items is executed.

[0072] The present application also provides a computer program product. When the computer program product is executed by a computer device, the method described in any of the preceding items is executed.

[0073] The present application also provides a computer device, the computer device comprising:

[0074] one or more processors;

[0075] a memory for storing one or more computer programs;

[0076] When the one or more computer programs are executed by the one or more processors, the one or more processors are caused to implement the method as described in any of the preceding items.

[0077] Figure 6 An exemplary system that can be used to implement various embodiments described in this application is shown;

[0078] like Figure 6 In some embodiments shown, the system 300 can be used as any of the devices in the various described embodiments. In some embodiments, the system 300 may include one or more computer-readable media (e.g., system memory or NVM / storage device 320) with instructions and one or more processors (e.g., (one or more) processors 305) coupled to the one or more computer-readable media and configured to execute instructions to implement modules to perform the actions described in this application.

[0079] For one embodiment, system control module 310 may include any suitable interface controller to provide any suitable interface to at least one of processor(s) 305 and / or any suitable device or component in communication with system control module 310 .

[0080] The system control module 310 may include a memory controller module 330 to provide an interface to the system memory 315. The memory controller module 330 may be a hardware module, a software module, and / or a firmware module.

[0081] The system memory 315 may be used, for example, to load and store data and / or instructions for the system 300. For one embodiment, the system memory 315 may include any suitable volatile memory, such as a suitable DRAM. In some embodiments, the system memory 315 may include double data rate type four synchronous dynamic random access memory (DDR4 SDRAM).

[0082] For one embodiment, system control module 310 may include one or more input / output (I / O) controllers to provide interfaces to NVM / storage device 320 and communication interface(s) 325 .

[0083] For example, NVM / storage device 320 may be used to store data and / or instructions. NVM / storage device 320 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable non-volatile storage device(s) (e.g., one or more hard disk drives (HDDs), one or more compact disk (CD) drives, and / or one or more digital versatile disk (DVD) drives).

[0084] NVM / storage device 320 may include storage resources that are physically part of the device on which system 300 is installed, or it may be accessible to the device without being part of the device. For example, NVM / storage device 320 may be accessed over a network via communication interface(s) 325.

[0085] Communication interface(s) 325 may provide an interface for system 300 to communicate over one or more networks and / or with any other suitable devices. System 300 may wirelessly communicate with one or more components of a wireless network in accordance with any of one or more wireless network standards and / or protocols.

[0086] For one embodiment, at least one of the processor(s) 305 may be packaged together with the logic of one or more controllers of the system control module 310 (e.g., the memory controller module 330). For one embodiment, at least one of the processor(s) 305 may be packaged together with the logic of one or more controllers of the system control module 310 to form a system-in-package (SiP). For one embodiment, at least one of the processor(s) 305 may be integrated on the same die with the logic of one or more controllers of the system control module 310. For one embodiment, at least one of the processor(s) 305 may be integrated on the same die with the logic of one or more controllers of the system control module 310 to form a system on chip (SoC).

[0087] In various embodiments, the system 300 may be, but is not limited to: a server, a workstation, a desktop computing device, or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a tablet computer, a netbook, etc.). In various embodiments, the system 300 may have more or fewer components and / or a different architecture. For example, in some embodiments, the system 300 includes one or more cameras, a keyboard, a liquid crystal display (LCD) screen (including a touch screen display), a non-volatile memory port, multiple antennas, a graphics chip, an application specific integrated circuit (ASIC), and a speaker.

[0088] In addition to the methods and devices described in the above embodiments, the present application also provides a computer-readable storage medium, which stores computer code. When the computer code is executed, the method described in any of the preceding items is executed.

[0089] The present application also provides a computer program product. When the computer program product is executed by a computer device, the method described in any of the preceding items is executed.

[0090] The present application also provides a computer device, the computer device comprising:

[0091] one or more processors;

[0092] a memory for storing one or more computer programs;

[0093] When the one or more computer programs are executed by the one or more processors, the one or more processors are caused to implement the method as described in any of the preceding items.

[0094] It should be noted that the present application can be implemented in software and / or a combination of software and hardware, for example, can be implemented using an application specific integrated circuit (ASIC), a general purpose computer or any other similar hardware device. In one embodiment, the software program of the present application can be executed by a processor to implement the steps or functions described above. Similarly, the software program of the present application (including relevant data structures) can be stored in a computer-readable recording medium, for example, a RAM memory, a magnetic or optical drive or a floppy disk and similar devices. In addition, some steps or functions of the present application can be implemented using hardware, for example, as a circuit that cooperates with a processor to perform each step or function.

[0095] In addition, a part of the present application may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present application through the operation of the computer. Those skilled in the art should understand that the existence of computer program instructions in computer-readable media includes but is not limited to source files, executable files, installation package files, etc., and accordingly, the way in which computer program instructions are executed by a computer includes but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to the computer.

[0096] Communication media include media by which communication signals containing, for example, computer readable instructions, data structures, program modules, or other data are transmitted from one system to another. Communication media may include guided transmission media such as cables and wires (e.g., fiber optic, coaxial, etc.) and wireless (unguided transmission) media that can propagate energy waves, such as acoustic, electromagnetic, RF, microwave, and infrared. Computer readable instructions, data structures, program modules, or other data may be embodied as a modulated data signal in, for example, a wireless medium such as a carrier wave or similar mechanism such as embodied as part of spread spectrum technology. The term "modulated data signal" refers to a signal whose one or more characteristics are changed or set in such a manner as to encode information in the signal. Modulation may be analog, digital, or a hybrid modulation technique.

[0097] By way of example and not limitation, computer-readable storage media may include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data. For example, computer-readable storage media include, but are not limited to, volatile memory, such as random access memory (RAM, DRAM, SRAM); and non-volatile memory, such as flash memory, various read-only memories (ROM, PROM, EPROM, EEPROM), magnetic and ferromagnetic / ferroelectric memories (MRAM, FeRAM); and magnetic and optical storage devices (hard disks, magnetic tapes, CDs, DVDs); or other media now known or later developed that can store computer-readable information / data for use by a computer system.

[0098] Here, according to an embodiment of the present application, a device is included, which includes a memory for storing computer program instructions and a processor for executing the program instructions, wherein, when the computer program instructions are executed by the processor, the device is triggered to run the methods and / or technical solutions based on the aforementioned multiple embodiments of the present application.

[0099] It is obvious to those skilled in the art that the present application is not limited to the details of the above exemplary embodiments, and that the present application can be implemented in other specific forms without departing from the spirit or basic features of the present application. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-restrictive, and the scope of the present application is limited by the attached claims rather than the above description, so it is intended to include all changes that fall within the meaning and scope of the equivalent elements of the claims in the present application. Any figure mark in the claims should not be regarded as limiting the claims involved. In addition, it is obvious that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the device claim can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any particular order.

Claims

1. A method for determining motion imaging delay, in, The method comprises: Acquiring a plurality of image information captured by a shooting device according to a preset shooting frequency for a display screen of the augmented reality device during a uniform motion of the shooting device and the augmented reality device, wherein the shooting device and the augmented reality device correspond to the same motion speed and motion direction, the augmented reality device locates a target object in a current real scene, and overlays and displays corresponding virtual presentation information according to the real-time current position of the target object on the display screen; The motion imaging delay of the augmented reality device is determined based on the moving distance information corresponding to the first image information and the second image information obtained by shooting the virtual presentation information before and after, the relative distance information corresponding to the target object and the virtual presentation information in the second image information, and the display frame duration information of the display screen, wherein the first image information and the second image information correspond to different display frames of the display screen.

2. The method according to claim 1, in, The shooting frequency is greater than the refresh frequency corresponding to the display screen.

3. The method according to claim 1, in, The photographing device and the augmented reality device are fixed on a mechanical arm, and the photographing device and the augmented reality device are synchronously moved at a uniform speed by driving the mechanical arm to move at a uniform speed.

4. The method according to claim 1 or 3, in, The motion direction corresponding to the uniform motion process only involves a two-dimensional plane.

5. The method according to claim 1, in, The target object includes any of the following: QR code; Barcode; Visual benchmark system code.

6. The method according to claim 1, in, The method further comprises: The first image information and the second image information are determined from the plurality of image information.

7. The method according to claim 6, in, The method further comprises: Acquire frame sequence number information of a display frame of the display screen corresponding to each piece of image information among the plurality of pieces of image information; Wherein, determining the first image information and the second image information from the plurality of image information includes: The first image information and the second image information are determined from the plurality of images according to the frame sequence number information.

8. The method according to claim 7, in, The display screen displays the frame sequence number information of the current display frame in real time.

9. The method according to claim 1, in, The method of determining the motion imaging delay of the augmented reality device according to the first image information and the moving distance information corresponding to the second image information obtained by shooting the virtual presentation information before and after, the relative distance information corresponding to the target object and the virtual presentation information in the second image information, and the display frame duration information of the display screen, comprises: The product of the ratio information of the relative distance information to the moving distance information and the frame duration information corresponding to the display frequency is calculated to obtain the motion imaging delay of the augmented reality device.

10. The method according to claim 1, in, The method further comprises: The moving distance information and the relative distance information are determined.

11. The method according to claim 10, in, The moving distance information and the relative distance information belong to pixel distance; Wherein, the determining the moving distance information and the relative distance information includes: The moving distance information and the relative distance information are determined by visually analyzing the target object and the virtual presentation information in the first image information and the second image information.

12. The method according to claim 10, in, The moving distance information and the relative distance information belong to physical distance; Wherein, the determining the moving distance information and the relative distance information includes: The moving distance information and the relative distance information are determined according to the measurement results obtained by measuring the target object and the virtual presentation information in the first image information and the second image information through a measuring tool.

13. The method according to claim 10, in, The method further comprises: Determine theoretical presentation position information corresponding to the virtual presentation information in the second image information according to the first position information corresponding to the target object in the second image information and the relative position relationship between the target object and the virtual presentation information on the display screen when the shooting device and the augmented reality device are in a static state before the uniform motion process; The relative distance information is determined according to the second position information corresponding to the virtual presentation information in the second image information and the theoretical presentation position information.

14. The method according to claim 10, in, When the shooting device and the augmented reality device are in a static state before being in the uniform motion process, the first display area of ​​the virtual presentation information on the display screen completely overlaps with the second display area of ​​the target object on the display screen; Wherein, the method further comprises: The relative distance information is determined according to the first position information corresponding to the target object in the second image information and the second position information corresponding to the virtual presentation information in the second image information.

15. The method according to claim 1, in, The method of determining the motion imaging delay of the augmented reality device according to the first image information and the moving distance information corresponding to the second image information obtained by shooting the virtual presentation information before and after, the relative distance information corresponding to the target object and the virtual presentation information in the second image information, and the display frame duration information of the display screen, comprises: Determine a motion imaging time delay of the second image information relative to the first image information according to movement distance information corresponding to the first image information and the second image information obtained by shooting the virtual presentation information before and after, and relative distance information corresponding to the target object and the virtual presentation information in the second image information, wherein the first image information and the second image information correspond to different display frames of the display screen; The motion imaging delay of the augmented reality device is determined according to the motion imaging delays respectively corresponding to the multiple groups of image information in the multiple image information.

16. A user equipment for determining motion imaging delay, in, The device comprises: A module for obtaining information of a plurality of images captured by a shooting device according to a preset shooting frequency during a uniform motion of the shooting device and the augmented reality device, wherein the shooting device and the augmented reality device correspond to the same motion speed and motion direction, and the augmented reality device locates a target object in a current real scene, and overlays and displays corresponding virtual presentation information according to the real-time current position of the target object on the display screen; Module one and module two are used to determine the motion imaging delay of the augmented reality device according to the corresponding moving distance information in the first image information and the second image information obtained by shooting the virtual presentation information before and after, the relative distance information between the target object and the virtual presentation information in the second image information, and the display frame duration information of the display screen, wherein the first image information and the second image information correspond to different display frames of the display screen.

17. A computer device for determining motion imaging delay, comprising a memory, a processor and a computer program stored in the memory, It is characterized in that The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 15.

18. A computer readable storage medium having a computer program / instructions stored thereon, It is characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 15 are implemented.

Images