Display method and device of extended display equipment and extended reality equipment

Abstract

The invention discloses a display method and device of an extended display device and the extended reality device, and belongs to the field of extended reality. The display method of the extended display equipment comprises the following steps: acquiring first displacement data of a fixation point of a user in an extended reality space; according to the first displacement data, an image taking frame of a preset size is moved in the first image, the overlapping area of the first image and the image taking frame is determined, and the first image is any frame of image of a video shot by the augmented reality device; and displaying the overlapping area of the first image.

Description

Technical Field

[0001] The present application belongs to the field of extended reality, and specifically relates to a display method and apparatus for an extended display device, an extended reality device, a readable storage medium, and a computer program product. Background Art

[0002] Extended Reality (XR) devices can provide an extended reality space, integrating virtual content with real scenes. Related technologies In order to avoid the shaking of the extended reality device and affect the recording effect of the real scene, optical image stabilization (OIS) and electronic image stabilization (EIS) are usually used for anti-shake processing. Optical image stabilization calculates the shaking of the device during shooting through a gyroscope, controls the movement of the lens for optical compensation, and achieves anti-shake. Electronic image stabilization is achieved by cropping the picture, taking part of the picture at the center of the photosensitive chip for output. When the device shakes during shooting, the cropping position is adjusted, the picture is compensated, and the picture is stable. However, when the user's gaze on the extended reality space provided by the extended reality device changes, the picture displayed by the extended reality device does not change, but the picture viewed by the user's eyes and formed in the brain changes. Frequent changes in gaze will cause the picture seen by the user's eyes to move frequently, which is easy to cause eye fatigue. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a display method, device and extended reality device for an extended display device, which can solve the problem of eye fatigue caused by frequent movement of the images seen by both eyes of the user.

[0004] In a first aspect, an embodiment of the present application provides a display method for an extended display device, the method comprising:

[0005] Acquire first displacement data of a gaze point of a user in an extended reality space;

[0006] Moving an imaging frame of a preset size in the first image according to the first displacement data to determine an overlapping area between the first image and the imaging frame, where the first image is any frame of a video captured by the augmented reality device;

[0007] The overlapping area of ​​the first image is displayed.

[0008] In a second aspect, an embodiment of the present application provides an extended reality device, including:

[0009] A first acquisition module, used to acquire first displacement data of a gaze point of a user in an extended reality space;

[0010] An image processing module, configured to move an imaging frame of a preset size in the first image according to the first displacement data, and determine an overlapping area between the first image and the imaging frame, wherein the first image is any frame of a video captured by the augmented reality device, and the size of the imaging frame is smaller than the size of the first image;

[0011] The display module is used to display the overlapping area of ​​the first image.

[0012] In a third aspect, an embodiment of the present application provides an extended reality device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be executed on the processor, and when the programs or instructions are executed by the processor, the steps of the method of the first aspect are implemented.

[0013] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method of the first aspect are implemented.

[0014] In a fifth aspect, an embodiment of the present application provides a chip, the chip including a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the method of the first aspect.

[0015] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method of the first aspect.

[0016] In an embodiment of the present application, in a display method of an extended display device, first displacement data of a user's gaze point in an extended real space is obtained; a capturing frame of a preset size is moved in the first image according to the first displacement data to determine an overlapping area between the first image and the capturing frame; and the overlapping area of ​​the first image is displayed so that the user can see the overlapping area. Displaying the overlapping area of ​​the first image can enable the display image to follow the movement of the user's eyes, thereby reducing jitter of the displayed image and reducing user eye fatigue. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a flow chart of a display method of an extended display device provided in an embodiment of the present application;

[0018] Figure 2 is a schematic diagram of a three-dimensional panoramic image provided in an embodiment of the present application;

[0019] Figure 3 is a schematic diagram of a plan map provided in an embodiment of the present application;

[0020] Figure 4 is a schematic diagram of an extended reality space provided in an embodiment of the present application;

[0021] Figure 5 is a schematic diagram of the structure of an extended reality device provided in an embodiment of the present application;

[0022] Figure 6 is a schematic diagram of the structure of an extended reality device provided in an embodiment of the present application;

[0023] Figure 7 It is a schematic diagram of the hardware structure of an extended reality device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0025] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0026] In the following, in combination with the accompanying drawings, the display method, device, extended reality device, readable storage medium and computer program product of the extended display device provided in the embodiments of the present application are described in detail through specific embodiments and their application scenarios.

[0027] See also Figure 1 , an embodiment of the present application provides a display method for an extended display device, the method comprising:

[0028] S110, obtaining first displacement data of a gaze point of a user in an extended reality space;

[0029] In the embodiments of the present disclosure, the extended reality device includes an XR (Extended Reality) device, an AR (Augmented Reality) device, an MR (Mixed Reality) device, and other devices that can provide an extended reality space. Virtual interactive objects can be displayed in the extended reality space, and the displayed virtual interactive objects include but are not limited to text objects, icon objects, three-dimensional model objects, and other types of interactive objects.

[0030] The extended reality device can obtain the user's gaze point through eye tracking technology. The user's gaze point in the extended reality space of the extended reality device can be obtained based on the user's eye displacement data, and the interactive focus can be controlled to move in the extended reality space based on the movement of the gaze point. The extended reality device can collect eye displacement data through a photosensitive sensor, and the photosensitive sensor can be an eye tracking camera. Eye displacement data includes but is not limited to eye movement direction, distance and speed. Exemplarily, the user's eyeball is fixed on point A in the extended reality space, and the photosensitive sensor emits modulated pulse infrared light. The pulse infrared light will be reflected after encountering the user's eyeball. The position and diameter of the user's pupil are inferred by detecting the reflected light, and then the user's eyeball movement direction and movement speed are determined according to the changes in the position and diameter of the pupil. According to the movement direction and movement speed of the eyeball, point A is determined as the gaze point in the extended reality space. When the user's eyeball moves from point A to point B in the extended reality space, the movement direction and movement speed of the eyeball change, and the gaze point is determined to move from point A to point B in the extended reality space according to the movement direction and movement speed of the eyeball.

[0031] The first displacement data is the displacement data of the gaze point in the extended real space, and the first displacement data can represent the direction, speed, distance, etc. of the gaze point. The first displacement data includes a displacement matrix corresponding to a plurality of continuous times.

[0032] S120, moving an imaging frame of a preset size in the first image according to the first displacement data, and determining an overlapping area between the first image and the imaging frame, where the first image is any frame of a video captured by the augmented reality device;

[0033] S130: Display the overlapping area of ​​the first image.

[0034] An image acquisition module may be provided in the extended reality device to capture the physical environment where the extended reality device is located, and obtain a video captured by the extended reality device, which may be referred to as a first video. The video includes a temporally continuous sequence of captured images captured by the image acquisition module, and the first image is one of the images in the sequence. The image acquisition module may be a fisheye camera module, and the first video is captured in real time by the fisheye camera module. The extended reality device may also perform image processing on the first video to obtain the desired video to be played, and the image display module of the extended reality device is used to display the image-processed video to the user. Image processing includes, but is not limited to, adding virtual objects, adjusting the color of the captured images, adjusting the clarity of the captured images, and the like.

[0035] The size of the image captured by the extended reality device is related to the image acquisition module in the extended reality device. Optionally, the size of the first image is 2460x2460. The size of the image frame can be set according to display needs. Since the image frame needs to move in the first image according to the first displacement data, the size of the image frame is smaller than the size of the first image. Optionally, the size of the image frame is 2260x2260, and the size of the overlapping area between the image frame and the first image is the same as the size of the image frame, and the size of the overlapping area is 2260x2260.

[0036] According to the first displacement data, a preset-size imaging frame is moved in the first image. It can be understood that the preset-size imaging frame is moved at the position of the first image based on the first displacement data, and the overlapping area is the area defined by the imaging frame for the first image. The overlapping area of ​​the first image is displayed by the image display module of the extended reality device, that is, the overlapping area in the first image can be rendered, and the area other than the overlapping area in the first image is not rendered, so that the user can view the overlapping area through the extended reality device. When the image captured by the extended reality device does not change, the overlapping area viewed by the user changes with the movement of the user's eyes. The extended reality device can process the captured images in the first video as the first image in sequence along the time sequence, obtain the overlapping areas corresponding to each captured image, and display the overlapping areas corresponding to each captured image in the time sequence, so that the user can watch the video that changes with the movement of the user's eyes, so as to reduce the image jitter viewed by the user due to the change of the user's gaze point.

[0037] For example, see Figure 2 , the user's gaze point A in the extended reality space is located at the midpoint of the display module, and the dotted line shows the user's gaze line. The overlapping area 22 displayed by the display module is an image that is centered on the captured first image 21, and the gaze point A is located at the center of the overlapping area 22. Please refer to Figure 3, the gaze point moves to the left from the midpoint of the display module. When the image displayed by the extended reality device remains unchanged, the image formed in the user's brain changes. Figure 2 Move the displayed position to Figure 3 The first displacement data of the position shown is used to determine the overlapping area 22 formed by the movement of the imaging frame in the first image 21. When the user observes the obtained overlapping area 22, the gaze point A is still located at the center of the overlapping area 22.

[0038] In the display method of the extended display device provided in the present application, first displacement data of the user's gaze point in the extended real space is obtained; a capturing frame of a preset size is moved in the first image according to the first displacement data to determine the overlapping area between the first image and the capturing frame; the overlapping area of ​​the first image is displayed so that the user can view the overlapping area, and the overlapping area is a part of the first image. Displaying the overlapping area of ​​the first image can enable the display image to follow the movement of the user's eyes, thereby reducing the jitter of the displayed image and reducing the user's eye fatigue.

[0039] In some embodiments, S120 includes:

[0040] S210, obtaining second displacement data of the extended reality device;

[0041] S220, moving an imaging frame of a preset size in the first image according to the second displacement data and the first displacement data;

[0042] S230: Determine an overlapping area between the first image and the imaging frame.

[0043] The second displacement data of the extended reality device can characterize the speed, distance, and direction of the overall displacement of the extended reality device. When the user shakes while wearing the extended reality device, the image acquisition module in the extended reality device also shakes, and the content displayed in the adjacent captured images captured by the image acquisition module changes greatly, causing the user to see a shaking picture generated by the captured image. According to the second displacement data and the first displacement data, a preset size of the imaging frame is moved in the first image, so that the position of the imaging frame combines the second displacement data and the first displacement data, and the picture viewing angle displayed in the overlapping area obtained by the imaging frame and the first image moves with the user's eyes and the extended reality device, thereby reducing the picture shaking in the overlapping area.

[0044] Optionally, the size of the image capturing frame and the size of the overlapping area are both preset sizes, and the image content in the image capturing frame of the first image can be directly rendered and displayed.

[0045] In one embodiment, the captured images in the first video all correspond to the second displacement data and the first displacement data, and the captured images can be processed in time sequence to obtain the overlapping areas corresponding to each captured image. Specifically, the position of the image frame in each captured image is determined according to the second displacement data and the first displacement data. The position of the image frame in the first image captured at time T can be based on the second displacement data and the first displacement data to determine the moving direction and distance of the image frame, and the position of the image frame in the first image captured at time T+1 can be determined according to the moving direction and distance to obtain the overlapping area between the first image captured at time T+1 and the image frame. The overlapping area of ​​the first image is displayed for the user to watch, so that the display angle of the second video viewed by the user follows the user's eyes and the movement of the extended reality device, thereby reducing the jitter of the displayed image and reducing the user's eye fatigue.

[0046] For example, see Figure 2 , in the first image 21 captured by the extended reality device at time T, the object B is located at the center of the first image 21. Figure 3 , the extended reality device moves to the left relative to object B. In the first image 21 captured at time T+1, object B is located in the right side of the first image 21. That is, the position of object B displayed in the first image 21 changes due to the movement of the extended reality device relative to object B. In the process of the extended reality device moving relative to object B, the user's gaze point A changes from Figure 2 The center shown moves to the lower left. In the extended reality device provided in the present application, the second displacement data of the extended reality device moving to the left is obtained, and the first displacement data of the gaze point A moving downward is obtained. The position of the image frame in the first image 21 is determined according to the second displacement data and the first displacement data. The image frame corresponding to the first image 21 taken at time T is located at the center of the first image 21, and the image frame corresponding to the first image 21 taken at time T+1 is located at the lower right side of the first image. The overlapping area 22 of the image frame and the first image 21 taken at time T+1 is displayed.

[0047] In some embodiments, the first displacement data includes a first inter-frame displacement matrix corresponding to the timestamp of the first image; the second displacement data includes a second inter-frame displacement matrix corresponding to the timestamp of the first image; S220 includes:

[0048] S310: moving an imaging frame of a preset size in the first image according to a second inter-frame displacement matrix corresponding to the timestamp of the first image and a first inter-frame displacement matrix corresponding to the timestamp of the first image.

[0049] The first video shot by the extended display device includes multiple captured images, and the second displacement data includes multiple inter-frame displacement matrices, any inter-frame displacement matrix corresponds to a first image, and the inter-frame displacement matrix corresponding to the captured image shot at time T represents the displacement of the extended display device from time T-1 to time T.

[0050] The extended display device captures the first image at time T, and the timestamp of the first image is time T. The second displacement data includes a second inter-frame displacement matrix corresponding to time T. The imaging frame is moved according to the second inter-frame displacement matrix corresponding to the first image and the first inter-frame displacement matrix corresponding to the timestamp of the first image, that is, the position of the imaging frame in the first image with the timestamp is determined by the first inter-frame displacement matrix and the second inter-frame displacement matrix with the same timestamp.

[0051] In some embodiments, S210 includes:

[0052] S410, acquiring a second image captured by the extended reality device, where the second image domain is an image captured one frame before the first image;

[0053] S420, extracting image features from the first image and the second image to obtain image features;

[0054] S430, obtaining a second inter-frame displacement matrix according to the displacement of the image features between the first image and the second image.

[0055] The first video of the extended reality device includes multiple captured images sorted along the shooting time, the first image and the second image are images with adjacent shooting times in the multiple captured images, and the shooting time of the second image is before the shooting time of the first image.

[0056] In this embodiment, the acquisition of the inter-frame displacement matrix can be achieved based on simultaneous localization and mapping (SLAM). In S420, feature detection is first performed on the first image and the second image. Feature detection includes but is not limited to detecting feature points using SIFT, SURF, and ORB algorithms to determine the image features that need to be tracked. Image features are usually areas in the image where local information changes greatly, such as a series of stable feature points such as corner points and edge points. These feature points will become the basis for subsequent displacement calculations. After acquiring the image features, the image features are tracked again, and the displacement of the image features in two adjacent frames is calculated, and the physical displacement required by the user to operate the extended reality device is filtered, thereby completing the motion estimation between adjacent first images, that is, obtaining the second inter-frame displacement matrix.

[0057] In this embodiment, the second inter-frame displacement matrix can be obtained by analyzing the first image, thereby simplifying the initial data required for calculation.

[0058] In some embodiments, S210 includes:

[0059] S510, acquiring a second image captured by the extended reality device, and obtaining third displacement data of the extended reality device based on the first image and the second image, wherein the second image is an image captured one frame before the first image;

[0060] S520, obtaining movement inertia data of the extended display device;

[0061] S530, correcting the third displacement data according to the movement inertia data to obtain second displacement data.

[0062] The extended reality device is provided with an inertial measurement unit (IMU) to collect mobile inertial data. The inertial measurement unit may be a gyroscope, an accelerometer, etc., and the mobile inertial data may include the angular velocity and acceleration of the extended reality device. The posture of the extended reality device may be inferred based on the mobile inertial data, thereby correcting the third displacement data to obtain more accurate second displacement data.

[0063] In this embodiment, the second displacement data is generated by analyzing the first image and the movement inertial data, so that the second displacement data is more accurate.

[0064] Optionally, clock synchronization is performed on the timestamp of the first image and the timestamp of the mobile inertial data; the synchronized mobile inertial data is converted into a relative posture of the extended reality device after integration; the relative posture is filtered to obtain a compensation posture, and the third displacement data is corrected according to the compensation posture to obtain the second displacement data.

[0065] In some embodiments, S310 includes:

[0066] S610, calculating the product of the first inter-frame displacement matrix and the second inter-frame displacement matrix to obtain a composite matrix;

[0067] S620: Move an imaging frame of a preset size in the first image according to the synthesis matrix.

[0068] The first inter-frame displacement matrix is ​​T1, and the second inter-frame displacement matrix is ​​T2, then the composite matrix T is:

[0069] T=T1*T2.

[0070] The first inter-frame displacement matrix represents the movement of the gaze point within the target time, and the second inter-frame displacement matrix represents the movement of the extended reality device within the target time. By calculating the product of the two matrices, the resulting composite matrix can describe the final transformation after the two movements. According to the composite matrix, the image frame of a preset size is moved in the first image, so that the image frame moves together with the movement of the user's gaze point and the movement of the extended reality device.

[0071] In some embodiments, after the imaging frame is moved according to the synthesis matrix, when at least a portion of the imaging frame is moved outside the first image, the edge of the imaging frame moved according to the synthesis matrix coincides with the edge of the first image, keeping the edge of the imaging frame within the first image.

[0072] The display method of the extended display device provided in the embodiment of the present application can be executed by an extended reality device. In the embodiment of the present application, the extended reality device provided in the embodiment of the present application is described by taking the extended reality device executing the display method of the extended display device as an example.

[0073] See also Figure 5 , Figure 5 is a schematic diagram of the structure of an extended reality device provided by another embodiment of the present application, such as Figure 5 As shown, the extended reality device may include:

[0074] A first acquisition module 501 is used to acquire first displacement data of a gaze point of a user in an extended reality space;

[0075] An image processing module 502 is used to move a picture frame of a preset size in the first image according to the first displacement data, and determine an overlapping area between the first image and the picture frame, wherein the first image is any frame of a video captured by the augmented reality device, and the size of the picture frame is smaller than the size of the first image;

[0076] The image display module 503 is used to display the overlapping area of ​​the first image.

[0077] In an embodiment of the present application, first displacement data of a user's gaze point in an extended reality space is obtained; a capturing frame of a preset size is moved in the first image according to the first displacement data to determine an overlapping area between the first image and the capturing frame; and the overlapping area of ​​the first image is displayed so that the user can view the overlapping area, which is a portion of the first image. Displaying the overlapping area of ​​the first image can enable the display image to follow the movement of the user's eyes, thereby reducing jitter in the displayed image and reducing eye fatigue in the user.

[0078] In an optional example, the image processing module 502 includes:

[0079] A first acquisition unit, used to acquire second displacement data of the extended reality device;

[0080] An image capturing frame moving unit, used for moving an image capturing frame of a preset size in the first image according to the second displacement data and the first displacement data;

[0081] The overlapping area calculation unit determines the overlapping area between the first image and the image capturing frame.

[0082] In an optional example, the first displacement data includes a first inter-frame displacement matrix corresponding to the timestamp of the first image; the second displacement data includes a second inter-frame displacement matrix corresponding to the first image; and the imaging frame moving unit is used to move an imaging frame of a preset size in the first image according to the second inter-frame displacement matrix corresponding to the first image and the first inter-frame displacement matrix corresponding to the timestamp of the first image.

[0083] In an optional example, the first acquisition unit is used to extract image features from the first image and the second image to obtain image features;

[0084] A second inter-frame displacement matrix is ​​obtained according to the displacement of the image features between the first image and the second image.

[0085] In an optional example, the first acquisition unit is used to acquire a second image captured by the extended reality device, and obtain third displacement data of the extended reality device based on the first image and the second image, where the second image is an image captured one frame before the first image;

[0086] Get the mobile inertial data of the extended display device;

[0087] The third displacement data is corrected according to the moving inertia data to obtain the second displacement data.

[0088] In an optional example, the image frame moving unit is used to calculate the product of the first inter-frame displacement matrix and the second inter-frame displacement matrix to obtain a composite matrix;

[0089] An imaging frame of a preset size is moved in the first image according to the synthesis matrix.

[0090] The extended reality device in the embodiment of the present application may be an extended reality device, or a component in the extended reality device, such as an integrated circuit or a chip. The extended reality device may be a terminal, or may be a device other than a terminal. Exemplarily, the extended reality device may be an augmented reality (AR), a virtual reality (VR) device, etc., which is not specifically limited in the embodiment of the present application.

[0091] The extended reality device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0092] The extended reality device provided in the embodiment of the present application can achieve Figures 1 to 4 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0093] Alternatively, if Figure 6 As shown, an embodiment of the present application further provides an extended reality device 100, including a processor 101 and a memory 102, wherein the memory 102 stores a program or instruction that can be executed on the processor 101, and when the program or instruction is executed by the processor 101, each step of the display method embodiment of the above-mentioned extended display device is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0094] It should be noted that the extended reality device in the embodiment of the present application includes the above-mentioned mobile extended reality device and non-mobile extended reality device.

[0095] Figure 7 A schematic diagram of the hardware structure of an extended reality device for implementing an embodiment of the present application.

[0096] The extended reality device 1000 includes but is not limited to: a radio frequency unit 1201, a network module 1202, an audio output unit 1203, an input unit 1204, a sensor 1205, a display unit 1206, a user input unit 1207, an interface unit 1208, a memory 1209, and a processor 1210 and other components.

[0097] Those skilled in the art will appreciate that the extended reality device 1000 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 1210 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Figure 7 The extended reality device structure shown in the figure does not constitute a limitation of the extended reality device. The extended reality device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be repeated here.

[0098] The processor 1210 is used to shoot the physical environment where the extended reality device is located to obtain a first video, where the first video includes first images of continuous frames;

[0099] The processor 1210 is configured to obtain first displacement data of a gaze point of a user in an extended reality space;

[0100] The processor 1210 is configured to move an imaging frame of a preset size in the first image according to the first displacement data, and determine an overlapping area between the first image and the imaging frame, wherein the first image is any frame of a video captured by the augmented reality device, and the size of the imaging frame is smaller than the size of the first image;

[0101] The processor 1210 is configured to display the overlapping area of ​​the first image.

[0102] In an embodiment of the present application, first displacement data of a user's gaze point in an extended reality space is obtained; a capturing frame of a preset size is moved in the first image according to the first displacement data to determine an overlapping area between the first image and the capturing frame; and the overlapping area of ​​the first image is displayed so that the user can view the overlapping area, which is a portion of the first image. Displaying the overlapping area of ​​the first image can enable the display image to follow the movement of the user's eyes, thereby reducing jitter in the displayed image and reducing eye fatigue in the user.

[0103] Optionally, the processor 1210 is used to obtain second displacement data of the extended reality device;

[0104] The processor 1210 is used to move an image frame of a preset size in the first image according to the second displacement data and the first displacement data;

[0105] The processor 1210 determines an overlapping area between the first image and the image capturing frame.

[0106] Optionally, the first displacement data includes a first inter-frame displacement matrix corresponding to the timestamp of the first image; the second displacement data includes a second inter-frame displacement matrix corresponding to the first image; the processor 1210 is used to move an imaging frame of a preset size in the first image according to the second inter-frame displacement matrix corresponding to the first image and the first inter-frame displacement matrix corresponding to the timestamp of the first image.

[0107] Optionally, the processor 1210 is used to extract image features from the first image and the second image to obtain image features;

[0108] The processor 1210 is configured to obtain a second inter-frame displacement matrix according to the displacement of the image features between the first image and the second image.

[0109] Optionally, the processor 1210 is used to obtain a second image captured by the extended reality device, and obtain third displacement data of the extended reality device based on the first image and the second image, where the second image is an image captured one frame before the first image;

[0110] The processor 1210 is used to obtain movement inertia data of the extended display device;

[0111] The processor 1210 is used to correct the third displacement data according to the movement inertia data to obtain the second displacement data.

[0112] Optionally, the processor 1210 is configured to calculate the product of the first inter-frame displacement matrix and the second inter-frame displacement matrix to obtain a composite matrix;

[0113] The processor 1210 is configured to move an imaging frame of a preset size in the first image according to the synthesis matrix.

[0114] It should be understood that in the embodiment of the present application, the input unit 1204 may include a graphics processor (Graphics Processing Unit, GPU) 12041 and a microphone 12042, and the graphics processor 12041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 1206 may include a display panel 12061, and the display panel 12061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1207 includes a touch panel 12071 and at least one of other input devices 12072. The touch panel 12071 is also called a touch screen. The touch panel 12071 may include two parts: a touch detection device and a touch controller. Other input devices 12072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0115] The memory 1209 can be used to store software programs and various data. The memory 1209 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 1209 may include a volatile memory or a non-volatile memory, or the memory 1209 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 1209 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0116] The processor 1210 may include one or more processing units; optionally, the processor 1210 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 1210.

[0117] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the display method embodiment of the above-mentioned extended display device is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0118] The processor is the processor in the extended reality device in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0119] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the various processes of the display method embodiment of the above-mentioned extended display device, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0120] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0121] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the display method embodiment of the extended display device as described above, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0122] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0123] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods of each embodiment of the present application.

[0124] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A display method for an extended display device, characterized in that: include: Acquire first displacement data of a gaze point of a user in an extended reality space; Moving an imaging frame of a preset size in the first image according to the first displacement data to determine an overlapping area between the first image and the imaging frame, wherein the first image is any frame of a video captured by an extended reality device, and the size of the imaging frame is smaller than that of the first image; The overlapping area of ​​the first image is displayed.

2. The display method of the extended display device according to claim 1, characterized in that: The step of moving an imaging frame of a preset size in the first image according to the first displacement data to determine an overlapping area between the first image and the imaging frame includes: Acquire second displacement data of the extended reality device; moving an imaging frame of a preset size in the first image according to the second displacement data and the first displacement data; An overlapping area between the first image and the imaging frame is determined.

3. The display method of the extended display device according to claim 2, characterized in that: The first displacement data includes a first inter-frame displacement matrix corresponding to the timestamp of the first image; the second displacement data includes a second inter-frame displacement matrix corresponding to the timestamp of the first image; The step of moving an image frame of a preset size in the first image according to the second displacement data and the first displacement data comprises: An imaging frame of a preset size is moved in the first image according to a second inter-frame displacement matrix corresponding to the timestamp of the first image and a first inter-frame displacement matrix corresponding to the timestamp of the first image.

4. The display method of the extended display device according to claim 3, characterized in that: The acquiring the second displacement data of the extended reality device includes: Acquire a second image captured by the extended reality device, where the second image is an image captured one frame before the first image; Performing image feature extraction on the first image and the second image to obtain image features; The second inter-frame displacement matrix is ​​obtained according to the displacement of the image features between the first image and the second image.

5. The display method of the extended display device according to claim 3, characterized in that: The acquiring the second displacement data of the extended reality device includes: Acquire a second image captured by the extended reality device, and obtain third displacement data of the extended reality device based on the first image and the second image, wherein the second image is an image captured one frame before the first image; Acquiring movement inertia data of the extended display device; The third displacement data is corrected according to the movement inertia data to obtain second displacement data.

6. The display method of the extended display device according to claim 3, characterized in that: The step of moving an image frame of a preset size in the first image according to a second inter-frame displacement matrix corresponding to the first image and a first inter-frame displacement matrix corresponding to a timestamp of the first image comprises: Calculating the product of the first inter-frame displacement matrix and the second inter-frame displacement matrix to obtain a composite matrix; An imaging frame of a preset size is moved in the first image according to the synthesis matrix.

7. An extended reality device, characterized in that: include: A first acquisition module, used to acquire first displacement data of a gaze point of a user in an extended reality space; an image processing module, configured to move an imaging frame of a preset size in a first image according to the first displacement data, and determine an overlapping area between the first image and the imaging frame, wherein the first image is any frame of a video captured by the extended reality device, and the size of the imaging frame is smaller than that of the first image; The display module is configured to display the overlapping area of ​​the first image.

8. An extended reality device, characterized in that: It comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the display method of the extended display device as described in any one of claims 1 to 6 are implemented.

9. A readable storage medium, characterized in that: The readable storage medium stores a program or an instruction, and when the program or the instruction is executed by the processor, the steps of the display method of the extended display device according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The program product is stored in a storage medium, and the program product is executed by at least one processor to implement the steps of the display method of the extended display device according to any one of claims 1 to 6.

Images