Screening method, system, electronic device and readable storage medium

Abstract

The application discloses a screenshot method and system, an electronic device and a storage medium. The screenshot method is applied to a near-eye display device and a wearable device, receives a screenshot instruction, acquires a real picture shot by a camera of the near-eye display device, wherein the screenshot instruction is triggered to be generated in response to a shortcut screenshot operation signal on the wearable device, converts a projection visual angle of an original virtual display picture of the near-eye display device from a human eye visual angle to a camera visual angle, and obtains a target virtual display picture, and fuses the real picture and the target virtual display picture into a target screenshot picture combined with reality and virtuality. The application solves the technical problem that a screenshot picture of the near-eye display device has virtuality-real difference.

Description

Technical Field

[0001] This application belongs to the field of near-eye display technology and relates to a screenshot method, system, electronic device and readable storage medium. Background Technology

[0002] Currently, there are two problems with taking screenshots on near-eye display devices: First, the screen only displays the virtual part, so a simple screenshot can only capture the virtual part and cannot capture the combined virtual and real content that the human eye actually sees. Second, there is a difference in perspective between the human eye and the camera, which causes the virtual content and the real content in the screenshot to not overlap. This results in a difference between the virtual and real images in the screenshots taken by near-eye display devices, making it difficult to capture a screenshot that matches the real image. Summary of the Invention

[0003] The main purpose of this application is to provide a screenshot method that aims to solve the technical problem of the discrepancy between the real and virtual images in the existing screenshot technology.

[0004] To achieve the above objectives, this application provides a screenshot method applied to near-eye display devices, the screenshot method comprising:

[0005] Receive a screenshot command to obtain the real-world image captured by the camera of the near-eye display device, wherein the screenshot command is triggered in response to a quick screenshot operation signal on the wearable device;

[0006] The projection angle of the original virtual display image of the near-eye display device is converted from the human eye's perspective to the camera's perspective to obtain the target virtual display image;

[0007] The real-world image is merged with the target virtual display image to create a combined virtual and real target screenshot.

[0008] Optionally, the step of converting the projection angle of the original virtual display image of the near-eye display device from the human eye's perspective to the camera's perspective to obtain the target virtual display image includes:

[0009] Based on the perspective transformation matrix between the human eye view and the camera view, and the projection matrix of the original model view of the camera, determine the projection transformation matrix from the human eye view to the camera view.

[0010] Based on the projection transformation matrix, the original virtual display image is rendered off-screen to obtain the target virtual display image.

[0011] Optionally, before the step of determining the projection transformation matrix from the human eye's viewpoint to the camera's viewpoint based on the viewpoint transformation matrix between the human eye's viewpoint and the camera's viewpoint, and the camera's original model view projection matrix, the screenshot method includes:

[0012] Obtain the initial positional offset between the human eye and the camera;

[0013] Based on the position offset, determine the perspective transformation matrix between the human eye's perspective and the camera's perspective.

[0014] Optionally, the step of merging the real-world image with the target virtual display image to form a combined virtual and real target screenshot includes:

[0015] By performing eye tracking on the user wearing the near-eye display device, the user's first gaze position is determined;

[0016] Based on the first gaze position, a corresponding first partial virtual display screen is determined in the target virtual display screen;

[0017] By merging the first partial virtual display screen with the real screen, a partial screenshot combining virtual and real elements is obtained.

[0018] Optionally, the step of fusing the first partial virtual display image and the real image to obtain a partial screenshot image combining virtual and real elements includes:

[0019] If there are multiple display targets at the first gaze position, the local near-eye display image corresponding to the first gaze position is magnified;

[0020] Re-track the user's eye movements to determine the user's second gaze position;

[0021] Based on the second gaze position, a corresponding second partial virtual display screen is determined in the first partial virtual display screen;

[0022] By merging the second partial virtual display screen with the real screen, a partial screenshot combining virtual and real elements is obtained.

[0023] Optionally, after the step of merging the real-world image with the target virtual display image to form a combined virtual and real target screenshot, the screenshot method further includes the following steps:

[0024] The inverse view transformation matrix is ​​determined based on the second position offset between the human eye and the camera.

[0025] Based on the camera's current model view projection matrix and the inverse view transformation matrix, recover the camera's original model view projection matrix.

[0026] To achieve the above objectives, this application provides a screenshot method for wearable devices, the screenshot method comprising:

[0027] In response to a quick screenshot operation signal, a screenshot command is sent to the near-eye display device. The screenshot command instructs the near-eye display device to merge the real-world image captured by the camera with the target virtual display image into a combined virtual and real target screenshot image. The target virtual display image is obtained by converting the projection perspective of the original virtual display image of the near-eye display device from the human eye's perspective to the camera's perspective.

[0028] To achieve the above objectives, this application also provides a screenshot system, the screenshot system comprising:

[0029] A near-eye display device is used to receive a screenshot command, acquire the real-world image captured by the camera of the near-eye display device; convert the projection angle of the original virtual display image of the near-eye display device from the human eye's perspective to the camera's perspective to obtain a target virtual display image; and merge the real-world image with the target virtual display image to form a target screenshot image that combines virtual and real elements.

[0030] Wearable devices are used to send screenshot commands to near-eye display devices in response to quick screenshot operation signals.

[0031] Optionally, the near-eye display device is used for:

[0032] Receive a screenshot command to obtain the real-world image captured by the camera of the near-eye display device, wherein the screenshot command is triggered in response to a quick screenshot operation signal on the wearable device;

[0033] The projection angle of the original virtual display image of the near-eye display device is converted from the human eye's perspective to the camera's perspective to obtain the target virtual display image;

[0034] The real-world image is merged with the target virtual display image to create a combined virtual and real target screenshot.

[0035] Optionally, the near-eye display device is used for:

[0036] The step of converting the projection angle of the original virtual display image of the near-eye display device from the human eye's perspective to the camera's perspective to obtain the target virtual display image includes:

[0037] Based on the perspective transformation matrix between the human eye view and the camera view and the original model view projection matrix of the camera, a projection transformation matrix is ​​determined from the human eye view to the camera view. The perspective transformation matrix is ​​used to transform the projection perspective from the human eye view to the camera view.

[0038] Based on the projection transformation matrix, the original virtual display image is rendered off-screen to obtain the target virtual display image.

[0039] Optionally, the near-eye display device is used for:

[0040] Before the step of determining the projection transformation matrix from the human eye view to the camera view based on the view transformation matrix between the human eye view and the camera view, and the original model view projection matrix of the camera, the screenshot method includes:

[0041] Obtain the initial positional offset between the human eye and the camera;

[0042] Based on the position offset, determine the perspective transformation matrix between the human eye's perspective and the camera's perspective.

[0043] Optionally, the near-eye display device is used for:

[0044] The step of fusing the real-world image with the target virtual display image to form a target screenshot that combines the real and virtual elements includes:

[0045] By performing eye tracking on the user wearing the near-eye display device, the user's first gaze position is determined;

[0046] Based on the first gaze position, a corresponding first partial virtual display screen is determined in the target virtual display screen;

[0047] By merging the first partial virtual display screen with the real screen, a partial screenshot combining virtual and real elements is obtained.

[0048] Optionally, the near-eye display device is used for:

[0049] The step of fusing the first partial virtual display image and the real image to obtain a partial screenshot image that combines virtual and real elements includes:

[0050] If there are multiple display targets at the first gaze position, the local near-eye display image corresponding to the first gaze position is magnified;

[0051] Re-track the user's eye movements to determine the user's second gaze position;

[0052] Based on the second gaze position, a corresponding second partial virtual display screen is determined in the first partial virtual display screen;

[0053] By merging the second partial virtual display screen with the real screen, a partial screenshot combining virtual and real elements is obtained.

[0054] Optionally, the near-eye display device is used for:

[0055] After the step of merging the real-world image with the target virtual display image to form a combined real-virtual screenshot, the screenshot method further includes the following steps:

[0056] The inverse view transformation matrix is ​​determined based on the second position offset between the human eye and the camera.

[0057] Based on the camera's current model view projection matrix and the inverse view transformation matrix, recover the camera's original model view projection matrix.

[0058] Optionally, the wearable device is used for:

[0059] In response to a quick screenshot operation signal, a screenshot command is sent to the near-eye display device. The screenshot command instructs the near-eye display device to merge the real-world image captured by the camera with the target virtual display image into a combined virtual and real target screenshot image. The target virtual display image is obtained by converting the projection perspective of the original virtual display image of the near-eye display device from the human eye's perspective to the camera's perspective.

[0060] This application also provides an electronic device, the electronic device comprising: a memory, a processor, and a program of the screenshot method stored in the memory and executable on the processor, wherein when the program of the screenshot method is executed by the processor, it can implement the steps of the screenshot method as described above.

[0061] This application also provides a computer-readable storage medium storing a program implementing a screenshot method, wherein when the program is executed by a processor, it implements the steps of the screenshot method as described above.

[0062] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the screenshot method described above.

[0063] This application provides a screenshot method applied to a near-eye display device. The screenshot method includes receiving a screenshot command, acquiring a real-world image captured by the camera of the near-eye display device, wherein the screenshot command is triggered in response to a quick screenshot operation signal on a wearable device, converting the projection view of the original virtual display image of the near-eye display device from the human eye's viewpoint to the camera's viewpoint to obtain a target virtual display image, and then merging the real-world image with the target virtual display image to form a combined virtual-real target screenshot image. Because the screenshot command is triggered in response to a quick screenshot operation signal on the wearable device, quick screenshots are achieved. Furthermore, because the projection viewpoint can be converted from the human eye's viewpoint to the camera's viewpoint, the target virtual display image under the converted camera viewpoint is consistent with the original virtual display image under the human eye's viewpoint. This ensures that the target screenshot image obtained after merging the target virtual display image with the real-world image is perfectly aligned, solving the technical defect of virtual-real image discrepancies caused by the difference between the camera's viewpoint and the human eye's viewpoint, and improving the virtual-real overlap of screenshots from near-eye display devices. Attached Figure Description

[0064] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0065] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0066] Figure 1 This is a flowchart illustrating the first embodiment of the screenshot method of this application;

[0067] Figure 2 This is a flowchart illustrating the second embodiment of the screenshot method of this application;

[0068] Figure 3 This is a flowchart illustrating the third embodiment of the screenshot method of this application;

[0069] Figure 4 This is a system schematic diagram of an embodiment of the screenshot method of this application;

[0070] Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the screenshot method in this application embodiment.

[0071] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0072] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0073] Example 1

[0074] Reference Figure 1 This application provides a screenshot method applied to near-eye display devices. In a first embodiment of the screenshot method, the screenshot method includes:

[0075] Step S10: Receive a screenshot command and obtain the real-world image captured by the camera of the near-eye display device, wherein the screenshot command is triggered in response to a quick screenshot operation signal on the wearable device.

[0076] Step S20: The projection angle of the original virtual display image of the near-eye display device is converted from the human eye's perspective to the camera's perspective to obtain the target virtual display image;

[0077] Step S30: The real-world image and the target virtual display image are merged into a target screenshot image that combines the real and virtual elements.

[0078] In this embodiment, it should be noted that the near-eye display device can be an AR (Augmented Reality) device or a MR (Mixed Reality) device. The AR device can be AR glasses. The human eye's field of view can be determined by the position of the lenses on the near-eye display device. When the lens position changes, the human eye's field of view also changes. For example, when wearing the near-eye display device, the user adjusts the lens spacing so that their eyes are aligned with the center of the lenses. When the vertical distance between the user's eyes and the center of the lenses is 20 mm, the corresponding position of the human eye's field of view can be 20 mm behind the center of the lenses. The vertical distance between the user's eyes and the center of the lenses can vary depending on how tightly the user wears the near-eye display device. The camera's field of view can be the angle that the image sensor on the near-eye display device can detect, or it can be the range that the camera can capture. The wearable device can be a watch. The quick screenshot operation signal can be the user's continuous tapping of the watch.

[0079] As an example, steps S10 to S30 include: receiving a screenshot command and acquiring the real-world image captured by the camera of the near-eye display device; determining a projection transformation matrix from the human eye's perspective to the camera's perspective based on the change information between the human eye's viewpoint and the camera's viewpoint; determining a target virtual display image through the projection transformation matrix, wherein the projection transformation matrix is ​​used to perform off-screen rendering on the original virtual display image; and merging the target virtual display image with the real-world image to obtain a target screenshot image that combines virtual and real elements, wherein the size of the target screenshot image can be consistent with the size of the image displayed by the near-eye display device. The specific implementation steps of the screenshot command sent by the wearable device can be referred to the specific implementation process in step X10 below, and will not be repeated here.

[0080] The step of converting the projection angle of the original virtual display image of the near-eye display device from the human eye's perspective to the camera's perspective to obtain the target virtual display image includes:

[0081] Step A10: Determine the projection transformation matrix from the human eye viewpoint to the camera viewpoint based on the viewpoint transformation matrix between the human eye viewpoint and the camera viewpoint, as well as the original model view projection matrix of the camera.

[0082] Step A20: Based on the projection transformation matrix, perform off-screen rendering on the original virtual display image to obtain the target virtual display image.

[0083] In this embodiment, it should be noted that the original model-view projection matrix of the camera is determined by the projection matrix, view matrix, and model matrix corresponding to the camera viewpoint before the human eye's perspective is converted to the camera's perspective. The model matrix transforms vertices from the local coordinate system to the world coordinate system, the view matrix transforms vertices from the world coordinate system to the view coordinate system, and the projection matrix transforms vertices from the view coordinate system to a normalized cube. The original model-view projection matrix of the camera can be used to map points in three-dimensional real space to two-dimensional space. The projection transformation matrix is ​​the matrix where the projection matrix has been converted from the human eye's perspective to the camera's perspective, and it is used to render the original virtual display image. Off-screen rendering refers to rendering without displaying the image on the screen; off-screen rendering does not affect the display image on the near-eye display device.

[0084] As an example, steps A10 to A20 include: determining the perspective transformation matrix between the human eye's viewpoint and the camera's viewpoint based on the positional difference between the human eye and the camera; obtaining the original projection matrix, view matrix, and model matrix corresponding to the camera's viewpoint; deriving the original model projection matrix of the camera based on the projection matrix, view matrix, and model matrix; multiplying the perspective transformation matrix and the original projection model matrix to obtain the projection transformation matrix; and recreating a rendering area on the projection transformation matrix to perform off-screen rendering of the original virtual display image to obtain the target virtual display image. Off-screen rendering can be performed by recreating an FBO (Frame Buffer Object) rendering area to render the original virtual display image off-screen. When rendering in the FBO rendering area, the image is not rendered onto the screen but into the off-screen buffer. In this embodiment, off-screen rendering of the original virtual display image using the projection transformation matrix ensures that the re-rendered target virtual display image is not displayed on the near-eye display device, thus guaranteeing that the near-eye display image is unaffected by the projection perspective transformation.

[0085] Prior to the step of determining the projection transformation matrix from the human eye's viewpoint to the camera's viewpoint based on the viewpoint transformation matrix between the human eye's viewpoint and the camera's viewpoint, and the camera's model view projection matrix, the screenshot method includes:

[0086] Step B10: Obtain the first position offset between the camera and the human eye;

[0087] Step B20: Determine the perspective transformation matrix between the human eye's perspective and the camera's perspective based on the position offset.

[0088] In this embodiment, it should be noted that the camera viewpoint can be a camera coordinate system with the camera's location as the origin, establishing the U-axis, V-axis, and N-axis. The human eye viewpoint can be a human eye coordinate system with the human eye's location as the origin, establishing the X-axis, Y-axis, and Z-axis. The first position offset can include a first distance offset and a first angle offset. The first distance offset can be the change in distance between each coordinate axis of the camera coordinate system and each coordinate axis of the human eye coordinate system. For example, the first distance offset can include a first vertical distance offset, a first horizontal distance offset, and a first front-back distance offset. The first angle offset can be the change in angle between each coordinate axis of the camera coordinate system and each coordinate axis of the human eye coordinate system. The first angle offset can be a first front-back angle offset, a second front-back angle offset, a third front-back angle offset, a first horizontal angle offset, a second horizontal angle offset, a third horizontal angle offset, a first vertical angle offset, a second vertical angle offset, and a third vertical angle offset, etc. The first distance offset and the first angle offset can be vectors, and the viewpoint transformation matrix can include a translation matrix and a rotation matrix.

[0089] As an example, steps B10 to B20 include: obtaining a first distance offset and a first angle offset from the camera viewpoint to the human eye viewpoint; determining a translation matrix based on the first distance offset and a rotation matrix based on the first angle offset; and generating a viewpoint transformation matrix from the camera viewpoint to the human eye viewpoint based on the translation matrix and the rotation matrix.

[0090] As an example, the specific process of determining the perspective transformation matrix between the human eye's viewpoint and the camera's viewpoint based on the first position offset is as follows:

[0091]

[0092] Where M view R is the viewpoint transformation matrix. view Let T be the rotation matrix. view The translation matrix is ​​used. The camera coordinate system includes the U-axis, V-axis, and N-axis, while the human eye coordinate system includes the X-axis, Y-axis, and Z-axis. The U-axis and X-axis have the same direction, the V-axis and Y-axis have the same direction, and the Z-axis and N-axis have the same direction. In the embodiment of this application, T... x T is the first front-to-back distance offset between the U-axis and the X-axis. y T is the first horizontal distance offset between the V-axis and the Y-axis. z U is the first vertical distance offset between the Z-axis and the N-axis. xU is the first forward and backward angular offset from the U-axis to the X-axis. y This is the second forward and backward angular offset from the U-axis to the Y-axis, U z V is the third forward and backward angular offset from the U-axis to the Z-axis. x V is the first horizontal angular offset from the V-axis to the X-axis. y The second horizontal angular offset from the V-axis to the Y-axis, V z N is the third horizontal angular offset from the V-axis to the Z-axis. x The first vertical angular offset from the N-axis to the X-axis, N y The second vertical angular offset from the N-axis to the Y-axis, N z The third vertical angular offset from the N-axis to the Z-axis is given. The translation matrix T can be determined by obtaining the first forward / backward distance offset, the first horizontal distance offset, and the first vertical distance offset. view By obtaining the first forward / backward angle offset, the second forward / backward angle offset, the third forward / backward angle offset, the first horizontal angle offset, the second horizontal angle offset, the third horizontal angle offset, the first vertical angle offset, the second vertical angle offset, and the third vertical angle offset, the rotation matrix R can be determined. view Translate matrix T view and rotation matrix R view Multiplying them together yields the viewpoint transformation matrix.

[0093] The screenshot method further includes the following steps after the step of merging the real-world image with the target virtual display image to form a combined real-virtual screenshot:

[0094] Step S40: Obtain the inverse view transformation matrix based on the second position offset between the human eye and the camera;

[0095] Step S50: Restore the original model view projection matrix of the camera based on the current model view projection matrix and the inverse view transformation matrix.

[0096] In this embodiment, it should be noted that the second positional offset between the human eye and the camera may include a second distance offset and a second angle offset. The second positional offset may include a second distance offset and a second angle offset. The second distance offset can be the change in distance between each coordinate axis of the human eye coordinate system and each coordinate axis of the camera coordinate system. For example, the second distance offset may include a second vertical distance offset, a second horizontal distance offset, and a second forward / backward distance offset. The second angle offset can be the change in angle between each coordinate axis of the human eye coordinate system and each coordinate axis of the camera coordinate system. The second angle offset may be a fourth forward / backward angle offset, a fifth forward / backward angle offset, a sixth forward / backward angle offset, a fourth horizontal angle offset, a fifth horizontal angle offset, a sixth horizontal angle offset, a fourth vertical angle offset, a fifth vertical angle offset, and a sixth vertical angle offset, etc. The second distance offset and the second angle offset may be vectors, and the inverse viewpoint transformation matrix may include an inverse translation transformation matrix and an inverse rotation transformation matrix.

[0097] As an example, steps S40 to S50 include: obtaining a second distance offset and a second angle offset between the human eye and the camera; determining an inverse translation transformation matrix and an inverse rotation transformation matrix based on the second distance offset; generating an inverse view transformation matrix between the human eye's viewpoint and the camera's viewpoint based on the inverse translation transformation matrix and the inverse rotation transformation matrix; and multiplying the camera's current model view projection matrix and the inverse view transformation matrix to restore the camera's original model view projection matrix.

[0098] As an example, the specific process of determining the inverse view transformation matrix between the human eye's viewpoint and the camera's viewpoint based on the second position offset is as follows:

[0099]

[0100] In the embodiments of this application, S view Let Q be the inverse view transformation matrix. view Let P be the inverse rotation transformation matrix. view The translation inverse transformation matrix is ​​used. The camera coordinate system includes the U-axis, V-axis, and N-axis, while the human eye coordinate system includes the X-axis, Y-axis, and Z-axis. In the embodiments of this application, T... u T is the second front-to-back distance offset between the X-axis and the U-axis. v T is the second horizontal distance offset between the Y-axis and the V-axis. n X is the second vertical distance offset between the N-axis and the Z-axis. u The fourth angular offset from the X-axis to the U-axis, X v The fifth angular offset from the X-axis to the V-axis, X nThe sixth angular offset from the X-axis to the N-axis, Y u The fourth horizontal angular offset from the Y-axis to the U-axis, Y v The fifth horizontal angular offset from the Y-axis to the V-axis, Y n Z is the sixth horizontal angular offset from the Y-axis to the N-axis. u The fourth vertical angular offset from the Z-axis to the U-axis, Z v The fifth vertical angular offset from the Z-axis to the V-axis, Z n The sixth vertical angular offset from the Z-axis to the N-axis can be used to determine the inverse translation matrix P by obtaining the second forward / backward distance offset, the second horizontal distance offset, and the second vertical distance offset. view By obtaining the fourth, fifth, and sixth forward and backward angular offsets, the fourth, fifth, and sixth horizontal angular offsets, the fourth, fifth, and sixth horizontal angular offsets, the fourth, fifth, and sixth vertical angular offsets, the inverse rotation transformation matrix Q can be determined. view Transform the translation inverse matrix P view and rotation inverse transformation matrix Q view Multiplying them together yields the inverse perspective transformation matrix.

[0101] This application provides a screenshot method applied to a near-eye display device. The screenshot method includes receiving a screenshot command, acquiring a real-world image captured by the camera of the near-eye display device, wherein the screenshot command is triggered in response to a quick screenshot operation signal on a wearable device, converting the projection view of the original virtual display image of the near-eye display device from the human eye's viewpoint to the camera's viewpoint to obtain a target virtual display image, and then merging the real-world image with the target virtual display image to form a combined virtual-real target screenshot image. Because the screenshot command is triggered in response to a quick screenshot operation signal on the wearable device, quick screenshots are achieved. Furthermore, because the projection viewpoint can be converted from the human eye's viewpoint to the camera's viewpoint, the target virtual display image under the converted camera viewpoint is consistent with the original virtual display image under the human eye's viewpoint. This ensures that the target screenshot image obtained after merging the target virtual display image with the real-world image is perfectly aligned, solving the technical defect of virtual-real image discrepancies caused by the difference between the camera's viewpoint and the human eye's viewpoint, and improving the virtual-real overlap of screenshots from near-eye display devices.

[0102] Example 2

[0103] Furthermore, referring to Figure 2Based on the above embodiments of this application, in another embodiment of this application, the same or similar content as the above embodiments can be referred to the above description, and will not be repeated hereafter. Based on this, the step of fusing the real-world image with the target virtual display image into a combined virtual-real target screenshot image includes:

[0104] Step C10: Determine the first gaze position of the user by performing eye tracking on the user wearing the near-eye display device;

[0105] Step C20: Based on the first gaze position, determine the corresponding first partial virtual display screen in the target virtual display screen;

[0106] Step C30: Combine the first partial virtual display screen with the real screen to obtain a partial screenshot that combines virtual and real elements.

[0107] As an example, steps C10 to C30 include: tracking the eyes of a user wearing a near-eye display device using eye-tracking technology to detect the user's first gaze position; determining whether the gaze duration of the user's first gaze position exceeds the preset gaze duration threshold; if it does not exceed the threshold, merging the real-world image with the target virtual display image to form a combined virtual-real target screenshot; if it exceeds the threshold, determining a corresponding first partial virtual display image in the target virtual display image based on the first gaze position; merging the first partial virtual display image and the real-world image to obtain a combined virtual-real partial screenshot; wherein the size of the partial screenshot image can be adjusted to match the size of the display image of the near-eye display device.

[0108] The step of determining the corresponding first partial virtual display frame in the target virtual display frame based on the first gaze position may further include:

[0109] Based on the first gaze position, the outline of the object in the target virtual display screen at the first gaze position is cropped as the first partial virtual display screen. Alternatively, the screen can be cropped with the first gaze position as the center in the target virtual display screen as the first partial virtual display screen. The screen can be a circle with a radius of 1 cm centered on the first gaze position, or a quadrilateral with an area of ​​1 square centimeter centered on the first gaze position.

[0110] This application embodiment uses eye-tracking technology to determine the first gaze position of the user wearing the near-eye display device, and determines the first partial virtual display image in the target virtual reality scene based on the first gaze position, thereby determining a partial screenshot image that combines virtual and real elements. This ensures that the partial screenshot image is also a picture that overlaps virtual and real elements, and realizes automatic screenshot of the local area that the user is concerned about, so that the screenshot image matches the user's intention and improves the accuracy of screenshots of the near-eye display device.

[0111] Example 3

[0112] Furthermore, referring to Figure 3 Based on the above embodiments of this application, in another embodiment of this application, the same or similar content as the above embodiments can be referred to the above description, and will not be repeated hereafter. Based on this, the step of fusing the first partial virtual display image and the real image to obtain a partial screenshot image combining virtual and real elements includes:

[0113] Step D10: If there are multiple display targets at the first gaze position, then the local near-eye display image corresponding to the first gaze position is magnified;

[0114] Step D20: Re-track the user's eye movements to determine the user's second gaze position;

[0115] Step D30: Based on the second gaze position, determine the corresponding second partial virtual display screen in the first partial virtual display screen;

[0116] Step D40: Merge the second partial virtual display screen and the real screen to obtain a partial screenshot that combines virtual and real elements.

[0117] In this embodiment, it should be noted that the local near-eye display image is a local image corresponding to the first gaze position displayed on the near-eye display device.

[0118] As an example, steps D10 to D40 include: detecting whether multiple display targets exist at the first gaze position; if no multiple display targets exist, taking a screenshot of the virtual-real hybrid image corresponding to the first gaze position; if multiple display targets exist, enlarging the local near-eye display image corresponding to the first gaze position; re-tracking the user's eyeballs to detect the user's second gaze position; determining whether the gaze duration at the second gaze position exceeds a preset gaze duration threshold; if it does not exceed the gaze duration threshold, determining the virtual-real hybrid partial screenshot image corresponding to the first gaze position; if it exceeds the gaze duration threshold, determining the corresponding second local virtual display image based on the second gaze position in the first local display image; restoring the size of the second local virtual display image; and merging the restored second local virtual display image with the real image to obtain the virtual-real hybrid partial screenshot image. The presence of multiple display targets at the first local position can be detected using image recognition.

[0119] The step of determining the corresponding second partial virtual display screen in the first partial virtual display screen based on the second gaze position may further include:

[0120] Based on the second gaze position, the outline of the object in the target virtual display screen at the second gaze position is cropped as the second partial virtual display screen. Alternatively, the screen can be cropped from the target virtual display screen with the second gaze position as the center as the second partial virtual display screen. This screen can be a circle with a radius of 2 cm centered on the second gaze position, or a quadrilateral with an area of ​​2 square centimeters centered on the second gaze position.

[0121] In this embodiment, eye-tracking technology is used to further predict whether the user wearing the near-eye display device intends to observe the first gaze position more clearly. If so, the first gaze position is magnified and displayed to the user wearing the near-eye display device. The second gaze position is determined, and the second partial virtual display image is determined by the second gaze position and the first partial virtual display image. This results in a partial screenshot image that combines virtual and real elements corresponding to the second partial position. This ensures that the partial screenshot image corresponding to the second partial position is also virtual and real, and the partial screenshot image is accurate to the second gaze position. This also ensures that the screenshot image can always match the intention of the user wearing the near-eye display device, further improving the accuracy of partial screenshots of the near-eye display device.

[0122] Example 4

[0123] This application also provides a screenshot method for wearable devices, the screenshot method comprising:

[0124] Step X10: In response to the quick screenshot operation signal, a screenshot command is sent to the near-eye display device. The screenshot command is used to instruct the near-eye display device to merge the real image captured by the camera with the target virtual display image into a target screenshot image that combines the real and virtual elements. The target virtual display image is obtained by converting the projection angle of the original virtual display image of the near-eye display device from the human eye's perspective to the camera's perspective.

[0125] In this embodiment, it should be noted that the near-eye display device can be AR glasses. The human eye's viewing angle can be determined by the position of the lenses on the near-eye display device. When the lens position changes, the human eye's viewing angle also changes. For example, when wearing the near-eye display device, the user adjusts the lens spacing so that their eyes are aligned with the center of the lenses. When the vertical distance between the user's eyes and the center of the lenses is 20 mm, the corresponding position of the human eye's viewing angle can be 20 mm behind the center of the lenses. The vertical distance between the user's eyes and the center of the lenses can vary depending on how tightly the user wears the near-eye display device. The wearable device can be a watch. The quick screenshot operation signal can be the user's continuous tapping of the watch.

[0126] As an example, step X10 includes: receiving a quick screenshot operation signal instruction triggered on the wearable device; and sending the screenshot instruction to the near-eye display device via the wearable device. The quick operation instruction can be a user tapping the wearable device with their knuckles, and the signal can be identified by detecting whether the wearable device vibrates.

[0127] This application provides a screenshot method for wearable devices. In response to a quick screenshot operation signal, a screenshot command is sent to a near-eye display device. The screenshot command instructs the near-eye display device to merge a real-world image captured by a camera with a target virtual display image into a combined real-world screenshot. The target virtual display image is obtained by converting the projection perspective of the original virtual display image on the near-eye display device from the human eye's perspective to the camera's perspective. Because a quick screenshot command can be sent to the near-eye display device, the speed of the screenshot operation is ensured, thereby improving the speed of screenshotting on the near-eye display device. Therefore, this application embodiment improves the speed of screenshotting on near-eye display devices.

[0128] Example 5

[0129] Reference Figure 4 This application also provides a screenshot system, which includes:

[0130] The near-eye display device 10 is used to receive a screenshot command, acquire the real-world image captured by the camera of the near-eye display device; convert the projection angle of the original virtual display image of the near-eye display device from the human eye's perspective to the camera's perspective to obtain a target virtual display image; and merge the real-world image with the target virtual display image to form a target screenshot image that combines virtual and real elements.

[0131] Wearable device 20 is used to send a screenshot command to the near-eye display device in response to a quick screenshot operation signal.

[0132] Optionally, the near-eye display device 10 is used for:

[0133] Receive a screenshot command to obtain the real-world image captured by the camera of the near-eye display device, wherein the screenshot command is triggered in response to a quick screenshot operation signal on the wearable device;

[0134] The projection angle of the original virtual display image of the near-eye display device is converted from the human eye's perspective to the camera's perspective to obtain the target virtual display image;

[0135] The real-world image is merged with the target virtual display image to create a combined virtual and real target screenshot.

[0136] Optionally, the near-eye display device 10 is used for:

[0137] The step of converting the projection angle of the original virtual display image of the near-eye display device from the human eye's perspective to the camera's perspective to obtain the target virtual display image includes:

[0138] Based on the perspective transformation matrix between the human eye view and the camera view and the original model view projection matrix of the camera, a projection transformation matrix is ​​determined from the human eye view to the camera view. The perspective transformation matrix is ​​used to transform the projection perspective from the human eye view to the camera view.

[0139] Based on the projection transformation matrix, the original virtual display image is rendered off-screen to obtain the target virtual display image.

[0140] Optionally, the near-eye display device 10 is used for:

[0141] Before the step of determining the projection transformation matrix from the human eye view to the camera view based on the view transformation matrix between the human eye view and the camera view, and the original model view projection matrix of the camera, the screenshot method includes:

[0142] Obtain the initial positional offset between the human eye and the camera;

[0143] Based on the position offset, determine the perspective transformation matrix between the human eye's perspective and the camera's perspective.

[0144] Optionally, the near-eye display device 10 is used for:

[0145] The step of fusing the real-world image with the target virtual display image to form a target screenshot that combines the real and virtual elements includes:

[0146] By performing eye tracking on the user wearing the near-eye display device, the user's first gaze position is determined;

[0147] Based on the first gaze position, a corresponding first partial virtual display screen is determined in the target virtual display screen;

[0148] By merging the first partial virtual display screen with the real screen, a partial screenshot combining virtual and real elements is obtained.

[0149] Optionally, the near-eye display device 10 is used for:

[0150] The step of fusing the first partial virtual display image and the real image to obtain a partial screenshot image that combines virtual and real elements includes:

[0151] If there are multiple display targets at the first gaze position, the local near-eye display image corresponding to the first gaze position is magnified;

[0152] Re-track the user's eye movements to determine the user's second gaze position;

[0153] Based on the second gaze position, a corresponding second partial virtual display screen is determined in the first partial virtual display screen;

[0154] By merging the second partial virtual display screen with the real screen, a partial screenshot combining virtual and real elements is obtained.

[0155] Optionally, the near-eye display device 10 is used for:

[0156] After the step of merging the real-world image with the target virtual display image to form a combined real-virtual screenshot, the screenshot method further includes the following steps:

[0157] The inverse view transformation matrix is ​​determined based on the second position offset between the human eye and the camera.

[0158] Based on the camera's current model view projection matrix and the inverse view transformation matrix, recover the camera's original model view projection matrix.

[0159] Optionally, the wearable device 20 is used for:

[0160] In response to a quick screenshot operation signal, a screenshot command is sent to the near-eye display device. The screenshot command instructs the near-eye display device to merge the real-world image captured by the camera with the target virtual display image into a combined virtual and real target screenshot image. The target virtual display image is obtained by converting the projection perspective of the original virtual display image of the near-eye display device from the human eye's perspective to the camera's perspective.

[0161] The screenshot system provided in this application, employing the screenshot method described in the above embodiments, solves the technical problem of image distortion in screenshots taken from near-eye display devices. Compared with the prior art, the beneficial effects of the screenshot provided in this application are the same as those of the screenshot method described in the above embodiments, and other technical features of this screenshot system are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0162] Example 6

[0163] This application provides an electronic device, which may be a playback device. The electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the screenshot method in the above embodiment.

[0164] The following is for reference. Figure 5 The diagram illustrates a structural schematic of an electronic device suitable for implementing embodiments of the present disclosure. The electronic devices in the embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0165] like Figure 5 As shown, an electronic device may include a processing system (such as a central processing unit, graphics processing unit, etc.) that can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) or programs loaded from a storage system into random access memory (RAM). The RAM also stores various programs and data required for the operation of the electronic device. The processing system, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.

[0166] Typically, the following systems can be connected to the I / O interface: input systems including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output systems including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage systems including, for example, magnetic tapes, hard drives, etc.; and communication systems. Communication systems allow electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although electronic devices with various systems are shown in the figures, it should be understood that it is not required to implement or possess all of the systems shown. More or fewer systems may be implemented alternatively.

[0167] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication system, or installed from a storage system, or installed from a ROM. When the computer program is executed by a processing system, it performs the functions defined above in the methods of embodiments of this disclosure.

[0168] The electronic device provided in this application solves the technical problem of image distortion in near-eye display devices by employing the screenshot method described in Embodiment 1 above. Compared with the prior art, the beneficial effects of the screenshot provided in this application are the same as those of the screenshot method provided in the above embodiments, and other technical features of this screenshot system are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0169] It should be understood that various parts of this disclosure can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics may be combined in any suitable manner in one or more embodiments or examples.

[0170] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0171] Example 7

[0172] This embodiment provides a computer-readable storage medium having computer-readable program instructions stored thereon, which are used to execute the screenshot method in Embodiment 1 above.

[0173] The computer-readable storage medium provided in this application embodiment may be, for example, a USB flash drive, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, system, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: displacement state information, distance compensation information, and orientation compensation information, or any suitable combination thereof.

[0174] The aforementioned computer-readable storage medium may be included in an electronic device or may exist independently without being assembled into an electronic device.

[0175] The aforementioned computer-readable storage medium carries one or more programs that, when executed by an electronic device, cause the electronic device to: receive a screenshot command, acquire the real-world image captured by the camera of the near-eye display device, wherein the screenshot command is triggered in response to a quick screenshot operation signal on the wearable device; convert the projection angle of the original virtual display image of the near-eye display device from the human eye's perspective to the camera's perspective to obtain a target virtual display image; and merge the real-world image with the target virtual display image to form a combined virtual and real target screenshot image.

[0176] Alternatively, in response to a quick screenshot operation signal, a screenshot command is sent to the near-eye display device, wherein the screenshot command is used to instruct the near-eye display device to merge the real image captured by the camera with the target virtual display image into a target screenshot image that combines the real and virtual elements. The target virtual display image is obtained by converting the projection perspective of the original virtual display image of the near-eye display device from the human eye perspective to the camera perspective.

[0177] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0178] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0179] The modules described in the embodiments of this disclosure can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0180] The computer-readable storage medium provided in this application stores computer-readable program instructions for executing the above-described screenshot method, thus solving the technical problem of image distortion in screenshots taken from near-eye display devices. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the screenshot method provided in the above-described embodiments, and will not be repeated here.

[0181] Example 8

[0182] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the screenshot prediction method described above.

[0183] The computer program product provided in this application solves the technical problem of image distortion in screenshots taken from near-eye display devices. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the screenshot method provided in the above embodiments, and will not be repeated here.

[0184] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent scope of this application.

Claims

1. A screenshot method, characterized in that, The screenshot method, applied to near-eye display devices, includes: Receive a screenshot command to obtain the real-world image captured by the camera of the near-eye display device, wherein the screenshot command is triggered in response to a quick screenshot operation signal on the wearable device; Obtain the initial positional offset between the human eye and the camera; Based on the position offset, determine the perspective transformation matrix between the human eye's perspective and the camera's perspective; Based on the perspective transformation matrix between the human eye view and the camera view and the original model view projection matrix of the camera, a projection transformation matrix is ​​determined from the human eye view to the camera view. The perspective transformation matrix is ​​used to transform the projection perspective from the human eye view to the camera view. Based on the projection transformation matrix, the original virtual display image of the near-eye display device is rendered off-screen to obtain the target virtual display image; By recreating the rendering area of ​​the FBO frame buffer object, the original virtual display image is rendered off-screen, and the rendering result is not output to the display screen of the near-eye display device, but saved to the off-screen buffer area. The real-world image is merged with the target virtual display image to create a combined virtual and real target screenshot.

2. The screenshot method as described in claim 1, characterized in that, The step of fusing the real-world image with the target virtual display image to form a target screenshot that combines the real and virtual elements includes: By performing eye tracking on the user wearing the near-eye display device, the user's first gaze position is determined; Based on the first gaze position, a corresponding first partial virtual display screen is determined in the target virtual display screen; By merging the first partial virtual display screen with the real screen, a partial screenshot combining virtual and real elements is obtained.

3. The screenshot method as described in claim 2, characterized in that, The step of fusing the first partial virtual display image and the real image to obtain a partial screenshot image that combines virtual and real elements includes: If there are multiple display targets at the first gaze position, the local near-eye display image corresponding to the first gaze position is magnified; Re-track the wearer's eye to determine the wearer's second gaze position; Based on the second gaze position, a corresponding second partial virtual display screen is determined in the first partial virtual display screen; By merging the second partial virtual display screen with the real screen, a partial screenshot combining virtual and real elements is obtained.

4. The screenshot method as described in claim 1, characterized in that, After the step of merging the real-world image with the target virtual display image to form a combined real-virtual screenshot, the screenshot method further includes the following steps: The inverse view transformation matrix is ​​determined based on the second position offset between the human eye and the camera. Based on the camera's current model view projection matrix and the inverse view transformation matrix, recover the camera's original model view projection matrix.

5. A screenshot method, characterized in that, The screenshot method, applied to wearable devices, includes: In response to a quick screenshot operation signal, a screenshot command is sent to the near-eye display device. The screenshot command instructs the near-eye display device to merge the real-world image captured by the camera with the target virtual display image into a combined virtual and real target screenshot image. The target virtual display image is obtained by the near-eye display device first obtaining the first positional offset between the human eye and the camera, determining the perspective transformation matrix between the human eye's viewpoint and the camera's viewpoint based on the positional offset, then determining the projection transformation matrix from the human eye's viewpoint to the camera's viewpoint based on the perspective transformation matrix and the camera's original model view projection matrix, and performing off-screen rendering on the original virtual display image of the near-eye display device based on the projection transformation matrix. This is done by recreating the rendering area of ​​the FBO frame buffer object, and the rendering result is not output to the display screen of the near-eye display device but only saved to the off-screen buffer area.

6. A screenshot system, characterized in that, The screenshot system includes: A near-eye display device is configured to receive a screenshot command, acquire the real-world image captured by the device's camera; acquire a first positional offset between the human eye and the camera; determine a perspective transformation matrix between the human eye's viewpoint and the camera's viewpoint based on the positional offset; determine a projection transformation matrix from the human eye's viewpoint to the camera's viewpoint based on the perspective transformation matrix and the camera's original model view projection matrix, wherein the perspective transformation matrix is ​​used to transform the projection perspective from the human eye's viewpoint to the camera's viewpoint; perform off-screen rendering on the original virtual display image of the near-eye display device based on the projection transformation matrix to obtain a target virtual display image; perform off-screen rendering on the original virtual display image by recreating the rendering area of ​​the FBO frame buffer object, obtain the rendering result, and save it to the off-screen buffer area without outputting it to the display screen of the near-eye display device; and merge the real-world image with the target virtual display image to form a combined virtual and real target screenshot image. Wearable devices are used to send screenshot commands to near-eye display devices in response to quick screenshot operation signals.

7. An electronic device, characterized in that, The electronic device includes: At least one processor; and a memory communicatively connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the steps of the screenshot method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program that implements the screenshot method, which is executed by a processor to implement the steps of the screenshot method as described in any one of claims 1 to 5.

Images