Screen picture capturing and sharing method and head-mounted display equipment

Abstract

The invention relates to the technical field of graphic rendering, and provides a screen picture capturing and sharing method and a head-mounted display device, which are used for improving the quality of screen picture operation. For the display data of the 3D application on the main screen, according to width and height information of the virtual screen or the display screen of the external equipment, zooming texture coordinates of the display data of the 3D application so as to capture a middle area of the display data of the 3D application on the main screen, the aspect ratio of the scaled texture coordinates is the same as that of the virtual screen or the display screen of the external equipment, so that the target image flow of the middle area intercepted on the main screen is matched with the resolution of the virtual screen or the display screen of the external equipment, the problems of picture filling, picture stretching and the like are avoided, and the display quality of the picture on the main screen after operation is improved.

Description

Technical Field

[0001] The present application relates to the field of graphics rendering technology and provides a method for capturing and sharing screen images and a head-mounted display device. Background Art

[0002] A head-mounted display (HMD) is a device that uses virtual display technology to present virtual images to users. Typically, a head-mounted display device includes two display screens for presenting virtual images, one for the user's left eye and one for the right eye. Figure 1 As shown, the binocular display screen presents a 3D stereoscopic visual effect by displaying the same object from different perspectives.

[0003] When using a head-mounted display (HMD), you can capture (e.g., take screenshots and record screens) and share (e.g., cast the screen) the 3D images on the binocular display screen for 2D display. However, due to the discrepancy between the resolution of the HMD's monocular screen and the 2D display screen, the captured and shared 3D images may appear black or stretched when displayed in 2D, affecting the user's viewing experience.

[0004] Therefore, achieving high-quality 2D display of 3D images is an urgent problem to be solved. Summary of the Invention

[0005] The embodiments of the present application provide a method for capturing and sharing screen images and a head-mounted display device for improving the quality of 2D display of 3D images.

[0006] In a first aspect, an embodiment of the present application provides a method for capturing a screen image, which is applied to a head-mounted display device, comprising:

[0007] Start the 3D application;

[0008] In response to a capture instruction for a main screen, acquiring an initial image stream of the main screen, where the main screen is any one of the binocular display screens, the main screen is used to display the 3D application, and each initial image frame in the initial image stream includes display data of the 3D application;

[0009] generating a virtual screen of the initial image stream according to the capture instruction, and obtaining preset width and height information of the virtual screen;

[0010] For each initial image frame in the initial image stream, scaling the texture coordinates of the display data of the 3D application to an aspect ratio determined by the aspect information, so as to intercept a middle area of ​​the display data to generate a target image stream;

[0011] The target image stream is rendered onto the virtual screen.

[0012] The beneficial effect of the above technical solution is as follows: when a head-mounted display device displays a 3D application, since the two binocular display screens are processed in the same manner when operating the binocular display screens, any one of the binocular display screens is used as the primary screen, and based on the received capture instruction for the primary screen, a virtual screen is generated for capturing the display data of the 3D application on the primary screen. In this way, for each frame of the initial image containing the display data of the 3D application on the primary screen, the texture coordinates of the display data of the 3D application are scaled to an aspect ratio determined by the preset width and height information of the virtual screen, so as to capture the middle area of ​​the display data of the 3D application to generate a target image stream, and the target image stream is used to render the virtual screen. Because the scaled texture coordinates have the same aspect ratio as the virtual screen, the resolution of the captured target image is adapted to the resolution of the virtual screen, eliminating the need for operations such as screen padding and proportional stretching, thereby improving the display quality of the 2D image on the virtual screen.

[0013] In addition, the scaled texture coordinates can capture the middle area of ​​the display data of the 3D application, thereby retaining the main content of interest to the user on the main screen as much as possible, ensuring the capture accuracy of the main screen.

[0014] Optionally, when the display of the 3D application occupies multiple layers, and different layers correspond to different display data of the 3D application, scaling the texture coordinates of the display data of the 3D application to a ratio equal to the aspect ratio determined by the aspect information to intercept a middle area of ​​the display data to generate a target image stream includes:

[0015] Obtaining, according to the on-screen manner of the multiple layers, a fused image of the display data of the 3D application on the multiple layers in the initial image;

[0016] The texture coordinates of the fused image are scaled to an aspect ratio determined by the aspect information, so as to intercept a middle area of ​​the fused image to generate a target image stream.

[0017] The beneficial effect of the above technical solution is: when the display of the 3D application on the main screen occupies multiple layers, and different layers are used to display different display data, in order to fully display all the content on the main screen, it is necessary to obtain a fused picture of the display data on multiple layers, so that the 2D picture displayed on the virtual screen is consistent with the picture displayed on the main screen, thereby improving the accuracy of picture capture on the main screen. Furthermore, by scaling the texture coordinates of the fused picture to the aspect ratio of the virtual screen, the middle area of ​​the main screen is captured to generate a target image stream, so that the area of ​​the captured fused picture is adapted to the resolution of the virtual screen, thereby improving the display quality of the target image on the virtual screen.

[0018] Optionally, obtaining a fused image of display data of the 3D application on the multiple layers in the initial image according to a mode of displaying the multiple layers on the screen includes:

[0019] When the on-screen display mode is to display the multiple layers on the screen together, a second thread is used to directly obtain a fused image generated by the first thread based on the display data on the multiple layers; wherein the first thread is a rendering thread of the 3D application, and the second thread is a thread corresponding to the screen operation;

[0020] When the on-screen display mode is to display the multiple layers on the screen separately, the second thread is used to obtain the display data on each layer from the first thread, and the display data are merged to generate a merged image.

[0021] The beneficial effect of the above technical solution is that no matter whether multiple layers are displayed on the screen together or multiple layers are displayed on the screen separately, a fused picture of the display data on multiple layers occupied by the 3D application can be obtained, and different display data on multiple layers can be displayed simultaneously, thereby ensuring that the fused picture is consistent with the picture displayed on any screen on the binocular display screen, ensuring that when the main screen is captured subsequently, the display data of multiple layers can be captured, thereby improving the accuracy of capturing the 3D application picture on the main screen.

[0022] Optionally, the adopting the second thread to respectively obtain display data on each layer and fusing the display data to generate a fused image includes:

[0023] Creating a target texture object, where the target texture object is used to store the fused image;

[0024] Binding the target texture object to the target texture unit of the shader, and setting parameters of the target texture object;

[0025] Using a frame buffer object, the source texture of the display data on each layer is rendered to the target texture object in the shader to generate a fused image, and the frame buffer object is used for off-screen rendering.

[0026] The beneficial effects of the above technical solution are: for the situation where multiple layers are displayed on the screen separately, a target texture object is created for storing the fused picture of the display data on multiple layers, thereby realizing the fused display of different display data on multiple layers, and by using the frame buffer object to fuse the source texture of the display data on each layer, off-screen rendering of the fused picture is realized.

[0027] Optionally, the target image stream is generated in the following manner:

[0028] Calculating a texture coordinate scaling ratio according to the width and height information, wherein a value range of the texture coordinate scaling ratio is (0, 1);

[0029] Generating texture coordinates of vertices of two triangles according to the texture coordinate scaling ratio; wherein the two triangles form a rectangle, and the value range of the texture coordinates is [0, 1];

[0030] A first picture is rendered according to the texture coordinates of the vertices of the two triangles to generate a target image stream, wherein the first picture is the original display data of the 3D application or a fusion picture of the display data of the 3D application on multiple layers.

[0031] The beneficial effects of the above technical solution are: the texture coordinate scaling ratio is calculated through the preset width and height information of the virtual screen, and the texture coordinates of the vertices of the two triangles are generated according to the texture coordinate scaling ratio. In this way, when the fused picture of multiple layers is sampled based on the texture coordinates, the rendered target image can be adapted to the size of the virtual screen, thereby avoiding operations such as filling and stretching of the target image, and improving the quality of the 2D display of the target image.

[0032] Optionally, the formula for the texture coordinate scaling ratio is expressed as:

[0033] scale=(1.0fH / W) / 2.0f

[0034] The texture coordinates of the vertices of the two triangles are respectively expressed as: (0.0, scale), (0.0, 1.0f-scale), (1.0, 1.0-scale), (0.0, scale), (1.0, 1.0f-scale), (1.0, scale);

[0035] Wherein, f represents a floating point number, H represents the height of the secondary screen, and W represents the width of the secondary screen.

[0036] The beneficial effects of the above technical solution are: the above-mentioned method of calculating the texture coordinate scaling ratio can realize the interception of the middle picture on the main screen of the head-mounted display device, thereby ensuring that the fused picture retains more areas of interest; and, since the value range of the texture coordinates is [0,1], and considering that the current display of 2D images is generally rectangular, therefore, according to the texture scaling coordinate ratio, two triangles with a value range of [0,1] and which can form a rectangle are generated, thereby ensuring the normal display of the fused 2D picture.

[0037] Optionally, rendering the target image stream onto the virtual screen includes:

[0038] Setting the viewport size according to preset width and height information of the virtual screen, and setting a rendering cropping area of ​​the same size according to the viewport size;

[0039] For the current target image in the target image stream, clearing the color buffer of the previous target image and clearing the depth buffer of the previous target image according to a preset clear screen color;

[0040] Calling the target rendering program of the shader, obtaining the vertex arrays corresponding to the two triangles generated based on the width and height information and setting the pointer of the vertex array;

[0041] activating a target texture unit and binding the texture of the current target image to the target texture unit;

[0042] Setting the texture parameters of the current target image;

[0043] Binding the target texture unit to the sampler of the shader;

[0044] Using the pointer to read the vertex array to draw the geometry in the rendering clipping area;

[0045] The geometry is rendered in the sampler according to the target texture unit to render the current target image onto the virtual screen.

[0046] The beneficial effects of the above technical solution are: for the target image stream after texture coordinate scaling, the viewport and rendering cropping area are set according to the width and height information of the virtual screen, and the target image stream is also intercepted by the width and height information of the virtual screen. In this way, when the target rendering program is called, when the geometric figures on the rendering cropping area are rendered in the sampler of the shader according to the target texture unit bound to the texture of the current target image, the resolution of the current target image can be adapted to the resolution of the secondary screen, thereby realizing the normal display of the 3D content of the head-mounted display device on the secondary screen, avoiding problems such as screen filling and stretching, and improving the quality of 2D display.

[0047] In a second aspect, an embodiment of the present application provides a method for sharing a screen image, which is applied to a head-mounted display device, and the method includes:

[0048] Start the 3D application;

[0049] In response to a sharing instruction for a main screen, obtaining an initial image stream of the main screen, where the main screen is any one of the binocular display screens, the main screen is used to display the 3D application, and each initial image frame in the initial image stream includes display data of the 3D application;

[0050] According to the sharing instruction, obtaining the width and height information of the display screen of the external device;

[0051] For each initial image frame in the initial image stream, scaling the texture coordinates of the display data of the 3D application to a ratio that is the same as the aspect ratio determined by the aspect information, so as to intercept a middle area of ​​the display data to generate a target image stream;

[0052] The target image stream is sent to the external device, so that the external device renders the target image stream on the display screen.

[0053] The beneficial effect of the above technical solution is as follows: when the head-mounted display device displays a 3D application, since the two screens are processed in the same manner when operating the binocular display screens, any one of the binocular display screens is used as the main screen. Based on the received sharing instruction for the main screen, the width and height information of the display screen of the external device used to share the display data of the 3D application on the main screen is obtained. In this way, for each frame of the initial image containing the display data of the 3D application on the main screen, the texture coordinates of the display data of the 3D application are scaled to the aspect ratio determined by the width and height information of the external device display screen, so as to intercept the middle area of ​​the display data of the 3D application to generate a target image stream, and the target image stream is rendered by the external device on its own display screen. Because the scaled texture coordinates are the same as the aspect ratio of the external device display screen, the resolution of the captured target image is adapted to the resolution of the virtual screen, eliminating the need for operations such as screen padding and proportional stretching, thereby improving the display quality of the 2D image on the external device display screen.

[0054] In addition, the scaled texture coordinates can capture the middle area of ​​the display data of the 3D application, thereby retaining the main content of interest to the user on the main screen as much as possible, ensuring the sharing accuracy of the main screen.

[0055] Optionally, when the display of the 3D application occupies multiple layers, and different layers correspond to different display data of the 3D application, scaling the texture coordinates of the display data of the 3D application to a ratio equal to the aspect ratio determined by the aspect information to intercept a middle area of ​​the display data to generate a target image stream includes:

[0056] Obtaining, according to the on-screen manner of the multiple layers, a fused image of the display data of the 3D application on the multiple layers in the initial image;

[0057] The texture coordinates of the fused image are scaled to an aspect ratio determined by the aspect information, so as to intercept a middle area of ​​the fused image to generate a target image stream.

[0058] The beneficial effect of the above technical solution is: when the display of the 3D application on the main screen occupies multiple layers, and different layers are used to display different display data, in order to fully display all the content on the main screen, it is necessary to obtain a fused picture of the display data on multiple layers, so that the 2D picture displayed on the virtual screen is consistent with the picture displayed on the main screen, thereby improving the accuracy of picture capture on the main screen. Furthermore, by scaling the texture coordinates of the fused picture to the aspect ratio of the virtual screen, the middle area of ​​the main screen is captured to generate a target image stream, so that the area of ​​the captured fused picture is adapted to the resolution of the virtual screen, thereby improving the display quality of the 2D picture on the display screen of the external device.

[0059] Optionally, obtaining a fused image of display data of the 3D application on the multiple layers in the initial image according to a mode of displaying the multiple layers on the screen includes:

[0060] When the on-screen display mode is to display the multiple layers on the screen together, a second thread is used to directly obtain a fused image generated by the first thread based on the display data on the multiple layers; wherein the first thread is a rendering thread of the 3D application, and the second thread is a thread corresponding to the screen operation;

[0061] When the on-screen display mode is to display the multiple layers on the screen separately, the second thread is used to obtain the display data on each layer from the first thread, and the display data are merged to generate a merged image.

[0062] The beneficial effects of the above technical solution are: no matter whether multiple layers are displayed on the screen together or multiple layers are displayed on the screen separately, a fused picture of the display data on multiple layers occupied by the 3D application can be obtained, and different display data on multiple layers can be displayed simultaneously, thereby ensuring that the fused picture is consistent with the picture displayed on any screen on the binocular display screen, ensuring that when the main screen is shared subsequently, the display data of multiple layers can be shared, thereby improving the accuracy of sharing the 3D application screen on the main screen.

[0063] Optionally, the adopting the second thread to respectively obtain display data on each layer and fusing the display data to generate a fused image includes:

[0064] Creating a target texture object, where the target texture object is used to store the fused image;

[0065] Binding the target texture object to the target texture unit of the shader, and setting parameters of the target texture object;

[0066] Using a frame buffer object, the source texture of the display data on each layer is rendered to the target texture object in the shader to generate a fused image, and the frame buffer object is used for off-screen rendering.

[0067] The beneficial effects of the above technical solution are: for the situation where multiple layers are displayed on the screen separately, a target texture object is created for storing the fused picture of the display data on multiple layers, thereby realizing the fused display of different display data on multiple layers, and by using the frame buffer object to fuse the source texture of the display data on each layer, off-screen rendering of the fused picture is realized.

[0068] Optionally, the target image stream is generated in the following manner:

[0069] Calculating a texture coordinate scaling ratio according to the width and height information, wherein a value range of the texture coordinate scaling ratio is (0, 1);

[0070] Generating texture coordinates of vertices of two triangles according to the texture coordinate scaling ratio; wherein the two triangles form a rectangle, and the value range of the texture coordinates is [0, 1];

[0071] A first picture is rendered according to the texture coordinates of the vertices of the two triangles to generate a target image stream, wherein the first picture is the original display data of the 3D application or a fusion picture of the display data of the 3D application on multiple layers.

[0072] The beneficial effects of the above technical solution are: the texture coordinate scaling ratio is calculated by the width and height information of the external device display screen, and the texture coordinates of the vertices of the two triangles are generated according to the texture coordinate scaling ratio. In this way, when the fused picture of multiple layers is sampled based on the texture coordinates, the rendered target image can be adapted to the size of the virtual screen, thereby avoiding operations such as filling and stretching of the target image, and improving the quality of the 2D display of the target image.

[0073] Optionally, the formula for the texture coordinate scaling ratio is expressed as:

[0074] scale=(1.0fH / W) / 2.0f

[0075] The texture coordinates of the vertices of the two triangles are respectively expressed as: (0.0, scale), (0.0, 1.0f-scale), (1.0, 1.0-scale), (0.0, scale), (1.0, 1.0f-scale), (1.0, scale);

[0076] Wherein, f represents a floating point number, H represents the height of the secondary screen, and W represents the width of the secondary screen.

[0077] The beneficial effects of the above technical solution are: the above-mentioned method of calculating the texture coordinate scaling ratio can realize the interception of the middle picture on the main screen of the head-mounted display device, thereby ensuring that the fused picture retains more areas of interest; and, since the value range of the texture coordinates is [0,1], and considering that the current display of 2D images is generally rectangular, therefore, according to the texture scaling coordinate ratio, two triangles with a value range of [0,1] and which can form a rectangle are generated, thereby ensuring the normal display of the fused 2D picture.

[0078] In a third aspect, an embodiment of the present application provides a head-mounted display device, comprising a processor, a memory, a binocular display screen, and a communication interface, wherein the communication interface, the binocular display screen, the memory, and the processor are connected via a bus;

[0079] The communication interface is used to communicate with external devices;

[0080] The binocular display screen is used to display images corresponding to the left and right eyes;

[0081] The memory stores a computer program, and the processor executes the steps of any one of the screen capture methods or screen sharing methods according to the computer program.

[0082] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of any one of the above-mentioned screen capture methods or screen sharing methods are implemented.

[0083] The technical effects brought about by any implementation method in the second to fourth aspects can be referred to the technical effects brought about by the corresponding implementation methods in the first and second aspects, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0085] Figure 1 Schematic diagram of the binocular display screen of the VR device provided in an embodiment of the present application;

[0086] Figure 2A A schematic diagram of black border filling during 2D display provided in an embodiment of the present application;

[0087] Figure 2BA schematic diagram of screen stretching during 2D display provided in an embodiment of the present application;

[0088] Figure 3 A flowchart of a method for capturing a screen image provided in an embodiment of the present application;

[0089] Figure 4 A flow chart of a method for generating a target image stream provided in an embodiment of the present application;

[0090] Figure 5 Schematic diagram of two triangles corresponding to the target image stream;

[0091] Figure 6 A schematic diagram showing the size relationship between the target image and the main screen;

[0092] Figure 7 This is the flow chart of the initialization method of the EGL environment;

[0093] Figure 8 Flowchart of the rendering method for the target image stream;

[0094] Figure 9 A schematic diagram of a single layer display during 2D display provided in an embodiment of the present application;

[0095] Figure 10 A flow chart of another method for generating a target image stream provided in an embodiment of the present application;

[0096] Figure 11 A flowchart of a screen sharing method provided in an embodiment of the present application;

[0097] Figure 12 A flow chart of another method for generating a target image stream provided in an embodiment of the present application;

[0098] Figure 13A Schematic diagram of the screen projection effect of the main screen;

[0099] Figure 13B A diagram showing the projection effect of multiple layers on the main screen;

[0100] Figure 14 This is a structural diagram of the head-mounted display device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0101] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of the technical solutions of this application, but not all of them. Based on the embodiments described in this application document, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the technical solutions of this application.

[0102] Based on the exemplary embodiments shown in this application, all other embodiments obtained by persons of ordinary skill in the art without inventive effort are within the scope of protection of this application. In addition, although the disclosure in this application is presented based on one or several exemplary examples, it should be understood that each aspect of the disclosure can independently constitute a complete technical solution.

[0103] In addition, the terms "comprises" and "comprising" and any variations thereof are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to those components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0104] The term "module" as used in this application refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0105] The following summarizes the design concept of the embodiments of the present application in combination with application scenarios.

[0106] When using a head-mounted display device, you can capture and share the 3D images on the binocular display screens to achieve 2D display. Because the images displayed by the binocular display screens are relatively small, in order to save device memory and improve operation efficiency, in some embodiments, you can operate the image on one of the binocular display screens.

[0107] Typically, the monocular screen of a head-mounted display device is generally square or approximately square (e.g., the resolution of the monocular screen is 1832*1920), that is, the aspect ratio of the monocular screen is 1:1 or approximately 1:1, while the screen of a 2D display is generally rectangular, such as a screen with an aspect ratio of 16:9 (e.g., 720p (1280*720) or 1080p (1920*1080)). Since the resolution of the monocular screen of a head-mounted display device is inconsistent with that of the 2D display screen, there are two main 2D display solutions for 3D images. One is to capture or share the middle part of the 3D image and fill the two sides with black, such as Figure 2AAs shown, the immersion is poor; the other is to stretch the 3D image to achieve full screen display, such as Figure 2B As shown, the picture is distorted.

[0108] In view of this, an embodiment of the present application provides a method for capturing and sharing screen images. For a 3D application on a main screen, the preset width and height information of the virtual screen is obtained according to a received capture instruction for the main screen, or the width and height information of the display screen of an external device is obtained according to a received sharing instruction for the main screen. Then, according to the aspect ratio determined by the obtained width and height information, the texture coordinates of the display data of the 3D application on the main screen are scaled. The aspect ratio of the scaled texture coordinates is the same as that of the virtual screen or the display screen of the external device, thereby obtaining a target image stream containing the middle area of ​​the 3D image on the main screen that the user is interested in. Since the scaled texture coordinates are the same as the aspect ratio of the virtual screen or the display screen of the external device, the resolution of the target image is adapted to the resolution of the virtual screen or the display screen of the external device, which not only solves the problem of poor immersion caused by black filling, but also solves the problem of image distortion caused by proportional stretching, effectively improving the quality of 2D display of 3D images, thereby providing a more comfortable and immersive viewing experience.

[0109] It should be noted that the head-mounted display device in the embodiment of the present application can be a virtual reality device (Virtual Reality, VR), an augmented reality device (Augmented Reality, AR), or a mixed reality device (Mixed Reality, MR).

[0110] See also Figure 3 , is a screen image capture method provided in an embodiment of the present application, which is executed by a head-mounted display device and mainly includes the following steps:

[0111] S301: Start a 3D application.

[0112] In some embodiments, when the head-mounted display device displays a 3D application, the display data of the 3D application is simultaneously displayed on the binocular display screens corresponding to the left and right eyes.

[0113] S302: In response to a capture instruction for the main screen, an initial image stream of the main screen is acquired.

[0114] Generally, the images of 3D applications on the binocular display screens corresponding to the left and right eyes are different but the difference is small. In most cases, there is no need to capture the 3D images of the binocular display screens. Therefore, in order to save device memory and improve capture efficiency, the display data of any one of the binocular display screens (recorded as the main screen) can be captured.

[0115] In some embodiments, after the head-mounted display device receives a user-triggered capture instruction for the main screen, it obtains an initial image stream of the main screen, wherein each initial image frame in the initial image stream contains display data of the 3D application.

[0116] In some embodiments, the capture instruction for the main screen includes but is not limited to operations such as screenshots and screen recording.

[0117] S303: Generate a virtual screen of the initial image stream according to the capture instruction, and obtain preset width and height information of the virtual screen.

[0118] The virtual screen is used to display the display data of the 3D application captured from the main screen, and its width and height information is preset.

[0119] Specifically, when performing a capture operation such as screenshot or screen recording on the main screen, the head-mounted display device calls the MediaProjection and Surface functions to create a virtual screen, and uses the virtual screen as a secondary screen to display the screenshot or screen recording on the main screen.

[0120] Since the main screen is generally square or approximately square, and the virtual screen is generally rectangular, in order to ensure normal 2D display of the initial image stream on the main screen, the width and height information of the virtual screen may be obtained.

[0121] For example, if the head-mounted display device is running Android, the virtual screen's management class is obtained through the DisplayManager service. The DisplayManager.getDisplay method in that class is used to retrieve the virtual screen's display object. Because the display object represents the virtual screen's display properties and configuration, obtaining the virtual screen's display object also provides information about its width and height. This information can be used to configure the EGLsurface object and the OpenGL ES viewport.

[0122] S304: For each initial image frame in the initial image stream, scale the texture coordinates of the display data of the 3D application to the aspect ratio determined by the aspect information to capture the middle area of ​​the display data to generate a target image stream.

[0123] In some embodiments, a class inheriting from Presentation is created to create and manage the viewport displayed on the virtual screen. The SurfaceHolder.Callback interface implemented in this class can handle events such as the creation, modification, and destruction of Surface objects. This allows for timely notification and execution of corresponding operations when the virtual screen's Surface object is ready.

[0124] Since the aspect ratio of the main screen is 1:1 or approximately 1:1, the main screen of the head-mounted display device is square or approximately square, while the virtual screen is mostly rectangular. In order to enable the target image stream of the 3D application captured on the main screen to be rendered normally on the virtual screen, the texture coordinates of the display data of the 3D application on the main screen can be scaled according to the width and height information of the virtual screen. The aspect ratio of the scaled texture coordinates is the same as that of the virtual screen, thereby cutting out the middle area of ​​the display data of the 3D application on the main screen, so that the target image stream containing the cut-out middle area is adapted to the resolution of the virtual screen.

[0125] In some embodiments, the target image stream generation process is as follows: Figure 4 , mainly includes the following steps:

[0126] S3041: Calculate the texture coordinate scaling ratio based on the width and height information.

[0127] Assuming the width of the virtual screen is W and the height is H, the formula for the texture coordinate scaling ratio is:

[0128] scale=(1.0fH / W) / 2.0f Formula 1

[0129] Where f is a floating point number and scale has a value range of (0, 1). Since 1.0fH / W is the remaining height, the middle area of ​​the 3D application on the main screen is captured by dividing by 2.0f.

[0130] S3042: Generate texture coordinates of the vertices of two triangles according to the texture coordinate scaling ratio.

[0131] Among them, two triangles form a rectangle, and the value range of texture coordinates is [0, 1].

[0132] like Figure 5 As shown in the figure, it is a schematic diagram of two triangles, and the texture coordinates of the vertices of the two triangles are respectively expressed as: (0.0f, scale), (0.0f, 1.0f-scale), (1.0f, 1.0f-scale), (0.0f, scale), (1.0f, 1.0f-scale), (1.0f, scale).

[0133] S3043: Rendering a middle area of ​​the display data of the 3D application on the main screen according to the texture coordinates of the vertices of the two triangles to generate a target image stream.

[0134] For each frame of the initial image, the texture coordinates of the vertices of the two triangles are input into the vertex shader for geometric calculation, and the calculation results are passed to the fragment shader. The fragment shader generates multiple fragments by interpolating the vertices and samples the intermediate area texture from the display data of the 3D application contained in the initial image to render each fragment, thereby generating the corresponding target image.

[0135] like Figure 6 The figure shows the relationship between the size of the target image and the main screen. The target image is an image of the middle area captured from the display data of the 3D application on the main screen.

[0136] In an embodiment of the present application, the texture coordinate scaling ratio is calculated using the width and height information of the virtual screen, and the texture coordinates of the vertices of two triangles are generated based on the texture coordinate scaling ratio. In this way, when the display data of the 3D application on the main screen is sampled based on the texture coordinates, the intercepted target image can be adapted to the size of the virtual screen, thereby avoiding operations such as filling and stretching the target image, and improving the quality of the captured target image. In addition, the method of calculating the texture coordinate scaling ratio can achieve the interception of the middle area of ​​the display data of the 3D application on the main screen, thereby ensuring that the intercepted target image retains more areas of interest to the user. Since the value range of the texture coordinates is [0,1], and considering that the current display of 2D images is generally rectangular, two triangles with a value range of [0,1] and that can form a rectangle are generated based on the texture scaling coordinate ratio, thereby ensuring the normal display of the target image on the virtual screen.

[0137] S305: Render the target image stream onto the virtual screen.

[0138] Typically, when a head-mounted display device displays a 3D application, it typically starts a rendering thread for the 3D application (referred to as the first thread in this embodiment of the application). The first thread renders the display data of the 3D application on the main screen. The content captured on the main screen is rendered on the virtual screen. Therefore, a second thread can be started separately for the virtual screen.

[0139] In some embodiments, OpenGL ES is a cross-platform graphics library that needs to be associated with the local window system. On the Android platform, the Embedded Graphics Library (EGL) is the interface that connects OpenGL ES and the local window system. Therefore, before rendering to the virtual screen, the EGL environment must be initialized in the second thread.

[0140] See also Figure 7 , which is the initialization process of the EGL environment, mainly includes the following steps:

[0141] S701: Add a second thread to the Java virtual machine to access objects and methods of the Java layer.

[0142] In some embodiments, since the Java Native Interface (JNI) is used to access objects and methods of the Java layer later, the javaVm->AttachCurrentThread() method may be called to attach the second thread to the Java virtual machine.

[0143] S702: Create an initial surface object of the virtual screen, and create an active window object based on the initial surface object.

[0144] At the Java level, a surface object can render images to be displayed on the screen. Therefore, an initial surface object is created for the virtual screen. This initial surface object is passed to the virtual screen using the ANativeWindow_fromSurface() function to create a corresponding active window (ANativeWindow). The ANativeWindow object is an interface for interacting with the local window system and can be used to create surface objects (i.e., target surface objects) in the subsequent EGL environment.

[0145] S703: Acquire an EGL display object, and initialize the connection between EGL and the virtual screen according to the EGL display object.

[0146] In some embodiments, the eglGetDisplay() function is called with EGL_DEFAULT_DISPLAY as a parameter to obtain an EGL display object. The EGL Display object represents the connection between EGL and the local window system. Further, the eglInitialize() function is called with the EGL Display object and other parameters to initialize the connection between EGL and the virtual screen.

[0147] S704: Query the attribute value of the EGL configuration.

[0148] In some embodiments, the eglGetConfigAttrib() function is used to query attribute values ​​of the EGL configuration (Config), such as color depth, buffer type, etc., to ensure that the selected EGLConfig meets the rendering requirements of the application.

[0149] S705: Create an EGL content instance according to the attribute value, the EGL display object and the display data on the main screen.

[0150] In some embodiments, the eglCreateContext() function is called, and the EGLDisplay object, the property values ​​of EGLConfig, and the display data of the 3D application on the main screen in the first thread are passed in to create a new EGL content (Context) instance, where the EGLContext instance represents the state information of the OpenGL ES rendering context.

[0151] S706: Create a target surface object according to the attribute value, the EGL display object and the active window object.

[0152] In some embodiments, the eglCreateWindowSurface() function is called, and the EGLDisplay object, the property values ​​of EGLConfig, and the created ANativeWindow object are passed in to create a new EGLSurface object, where the EGLSurfac object is the target of the OpenGL ES rendering output, recorded as the target surface object.

[0153] S707: Bind the target surface object and the EGL content instance to the second thread, so that the second thread renders the target surface object.

[0154] In some embodiments, the eglMakeCurrent() function is called, and the EGLDisplay object, the property values ​​of EGLConfig, the EGLSurface object and the EGLContext instance are passed in, so that the EGLSurface object and the EGLContext instance are associated and bound to the second thread. In this way, the second thread can render the image on the virtual screen by calling the graphics API such as OpenGL ES to render the target surface object.

[0155] In an embodiment of the present application, the EGL environment is initialized by the second thread corresponding to the screen capture. Since EGL is an interface for creating rendering windows on a variety of graphics hardware, it enables graphics APIs such as OpenGL ES and OpenVG to work on different operating systems and hardware platforms, providing a unified underlying abstraction for graphics rendering. Therefore, after EGL is initialized, the second thread can render the virtual screen.

[0156] In some embodiments, after the EGL environment is initialized, the virtual screen rendering is achieved by drawing the intercepted target image stream onto the target surface object. Figure 8 , mainly includes the following steps:

[0157] S3051: Setting the viewport size according to the width and height information of the virtual screen, and setting a rendering cropping area of ​​the same size according to the viewport size.

[0158] In some embodiments, the glViewport(x, y, W, H) function is called to set the OpenGL viewport size, which determines the display area of ​​the target image stream on the target surface object of the virtual screen, and the glScissor(x, y, W, H) function is used to define a rendering clipping area. Only rendering operations within this area will be processed, and rendering outside of this area will be ignored.

[0159] S3052: For the current target image in the target image stream, clear the color buffer of the previous target image and clear the depth buffer of the previous target image according to a preset clearing color.

[0160] In some embodiments, the glClearColor(red, green, blue, alpha) function is called to set the color used to clear the color buffer, which will fill the entire color buffer when clearing operations between two adjacent frames of target images, and the glClear(mask) function is called, passing in GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT as parameters to clear the color buffer and depth buffer in preparation for the rendering operation of the next frame.

[0161] S3053: Call the target rendering program of the shader, obtain the vertex arrays corresponding to the two triangles generated based on the width and height information, and set the pointer of the vertex array.

[0162] In some embodiments, a shader program object is specified as a target rendering program using the glUseProgram(program) function, and the target rendering program performs rendering operations as part of the current rendering state.

[0163] To ensure the normal rendering of the target image stream on the virtual screen, two triangles are generated based on the width and height information of the virtual screen. The vertex arrays corresponding to the two triangles are passed in by calling glEnableVertexAttribArray(index), and then the pointer to the vertex array is set using glVertexAttribPointer(index, size, type, normalized, stride, pointer) to tell OpenGL how to interpret these vertex data.

[0164] S3054: Activate the target texture unit and bind the texture of the current target image to the target texture unit.

[0165] In some embodiments, the glActiveTexture(texture) function is called to activate a specific target texture unit so that subsequent texture binding operations can be bound to the correct texture unit, and the glBindTexture(target, texture) function is called to bind the texture of the current target image to the currently activated target texture unit, so that the texture of the current target image can be used during rendering.

[0166] S3055: Set the texture parameters of the current target image.

[0167] In some embodiments, the glTexParameteri(target, pname, param) function is called to set the parameters of the texture, such as the filtering mode and the wrapping mode, to control the rendering behavior of the texture.

[0168] S3056: Bind the target texture unit to the shader's sampler.

[0169] When rendering in the rendering pipeline, the shader's sampler is bound to the target texture unit through the glBindSampler(unit, sampler) function to affect the texture sampling process.

[0170] S3057: Using pointer to read vertex array to draw geometry in render clip region.

[0171] In some embodiments, the glDrawArrays(mode, first, count) function is called to draw geometric shapes, where the mode parameter specifies the drawing mode (such as GL_TRIANGLES), first is the starting index of the vertex array corresponding to the target texture unit, and count is the number of vertices to be drawn.

[0172] S3058: Rendering the geometry according to the target texture unit in the sampler to render the current target image onto the virtual screen.

[0173] When OpenGL draws, it allocates two buffers, front and back, to avoid flickering in the window during the drawing process. The front buffer is used for screen refresh, and the back buffer is used for drawing. When the eglSwapBuffers(display, surface) function is called, the back buffer and the front buffer of the target surface object are swapped, and the target surface object is rendered to the virtual screen.

[0174] For the target image stream captured based on the width and height information of the virtual screen, the viewport and the rendering cropping area are set according to the width and height information of the virtual screen. In this way, when the target rendering program is called and the geometry on the rendering cropping area is rendered in the sampler of the shader according to the target texture unit bound to the texture of the current target image, the resolution of the current target image can be adapted to the resolution of the virtual screen, thereby achieving normal display of the image captured on the main screen of the head-mounted display device on the virtual screen, avoiding problems such as image filling and stretching, and improving the quality of screen capture.

[0175] When a user wears a head-mounted display device for a 3D experience, the head-mounted display device can remind and warn the user through the prompt function or perspective function in the 3D application, and the display data of these functions and the display data of the 3D application correspond to different layers respectively. In the OpenXR framework, there are two main ways to display applications with multiple layers on the screen: one is that when multiple layers have the same frame rate, the first thread fuses the display data of multiple layers, performs distortion processing, and then renders them to the binocular display screen. In this case, since the images of multiple layers are synthesized and displayed on the screen, the display data of all layers can be captured; the other is that when multiple layers have different frame rates (such as the system frame rate is 72HZ, in order to reduce dizziness and delay, the screen frame rate of the layer corresponding to the 3D application is 72HZ, when the camera frame rate is 30HZ, the screen frame rate of the layer corresponding to the perspective function is 30HZ), the first thread dedistorts the display data of different layers and renders them to the binocular display screen. In this case, since the display data of multiple layers are displayed on the screen separately, only the display data of the main layer (the layer where the display data of the 3D application is located) is captured, and the display data on the layer corresponding to the prompt function or the perspective function cannot be captured or shared, resulting in inconsistent content between the main screen and the virtual screen, such as Figure 9 shown.

[0176] In some embodiments, when the display of a 3D application occupies multiple layers, and different layers correspond to different display data of the 3D application, each frame of the initial image in the acquired initial image stream includes display data of multiple layers. In this case, for the capture process of the main screen image, the method for generating the target image stream in S304 is as follows: Figure 10 , mainly includes the following steps:

[0177] S1001: Obtaining a fusion image of display data of a 3D application on multiple layers in an initial image according to a method of displaying multiple layers on the screen.

[0178] Because multiple layers are displayed differently, the capture results for the main screen are also different. To mitigate this effect, a second thread is invoked to capture the main screen. Through inter-thread communication and data transfer, the display data of multiple layers on the main screen is passed to the second thread, along with the surface object of the virtual screen. This allows the second thread to render the captured content on the main screen onto the virtual screen, ensuring synchronization between the virtual and main screen displays.

[0179] Specifically, the data acquisition process for different screen-up methods is as follows:

[0180] S1001_1: Determine the screen display mode for multiple layers. If they are displayed together, execute S1001_2. If they are displayed separately, execute S1001_3.

[0181] S1001_2: Use the second thread to directly obtain the fusion image generated by the first thread according to the display data on multiple layers.

[0182] Among them, the first thread is the rendering thread of the 3D application, and the second thread is the thread corresponding to the screen operation. Since the first thread has fused the display data of multiple layers when rendering the display data of multiple layers to the main screen, at this time, the second thread can directly obtain the fused screen of the display data of multiple layers generated by the first thread for subsequent operations.

[0183] S1001_3: Use the second thread to obtain display data on each layer from the first thread, and fuse the display data to generate a fused image.

[0184] Since the first thread renders the display data of multiple layers onto the main screen separately, at this time, in order to ensure the synchronization of the display content of the virtual screen and the main screen, the second thread can be used to obtain the display data of each layer from the first thread separately, and fuse the various display data to generate a fused screen for subsequent operations.

[0185] In some embodiments, the second thread's fusion process of display data of each layer mainly includes the following steps:

[0186] S1001_31: Create target texture object.

[0187] The target texture object is used to store the fused image.

[0188] S1001_32: Bind the target texture object to the target texture unit of the shader and set the parameters of the target texture object.

[0189] Since texture data from a texture object cannot be passed directly to a shader, texture objects can be bound to texture units. This allows the sampler in the shader to render using the corresponding texture unit. The parameters that can be set include filtering and wrapping modes.

[0190] S1001_33: Use the frame buffer object to render the source texture of the display data on each layer to the target texture object in the shader to generate a fused image.

[0191] Among them, the frame buffer object (Frame Buffer Object, FBO) is used for off-screen rendering, and data can be subsequently obtained from the frame buffer object to render the target texture object.

[0192] For the situation where multiple layers are displayed on the screen separately, a target texture object is created to store the fused image of multiple layers, so as to realize the fused display of different display data on multiple layers. Moreover, the source texture of the display data of each layer is fused by using the frame buffer object, so as to realize off-screen rendering of the fused image.

[0193] In some embodiments, the rendering process of the fused image is as follows:

[0194] Step 1: Create a frame buffer object and associate the frame buffer object with the target texture object.

[0195] Among them, the frame buffer object is used to store rendering results.

[0196] Step 2: Determine whether the frame buffer object is complete and configured correctly. If so, proceed to step 3; otherwise, return to step 1.

[0197] Step 3: Set the rendering area of ​​the target texture object to be consistent with the size of the source texture of the display data of each layer.

[0198] Step 4: Assign a texture unit to each layer and activate it.

[0199] Step 5: Input the source texture of the display data on each layer into the shader respectively, and render the frame buffer object according to the vertex information and texture coordinates corresponding to the currently activated texture unit.

[0200] The texture coordinates are texture coordinates of vertices of display data on a corresponding layer, and the vertex information includes information of vertices generated by vertex interpolation based on the corresponding display data.

[0201] In the rendering pipeline, shaders include vertex shaders and fragment shaders. Vertex shaders are used to calculate the vertices of graphics, and fragment shaders are used to display pixels. In fragment shaders, data from different textures can be mixed, superimposed, and other operations.

[0202] In some embodiments, when the source textures of display data of multiple layers are mixed and superimposed, they may be based on the preset levels corresponding to the respective layers.

[0203] Taking the 3D application prompt and perspective functions as an example, the layer corresponding to the prompt function has the highest level and is located at the top of the screen. The layer corresponding to the perspective function has the lowest level and is located at the bottom of the screen. The layer corresponding to the 3D application is in the middle layer. When mixing and superimposing, different transparency levels can be set between the layers to achieve synchronous display.

[0204] Step 6: Render the target texture object according to the data stored in the frame buffer object to generate a fused image.

[0205] For the situation where multiple layers are displayed on the screen separately, by allocating a texture unit to each layer, the source textures of the display data of multiple layers can be passed to the shader, thereby fully utilizing the powerful graphics rendering capabilities of the shader to render the frame buffer object, achieving efficient off-screen rendering, and then rendering the target texture object according to the fused image stored in the frame buffer object, realizing real-time rendering of the fused image.

[0206] S1002: Scaling the texture coordinates of the fused image to the aspect ratio determined by the aspect information to capture the middle area of ​​the fused image to generate a target image stream.

[0207] When the display of the 3D application on the main screen occupies multiple layers, and different layers are used to display different display data, in order to fully display all the content on the main screen, it is necessary to obtain a fused picture of the display data on multiple layers. The fused picture contains all the display data of multiple layers on the main screen. In this way, when the texture coordinates of the fused picture are subsequently scaled by the width and height information of the virtual screen, it can be ensured that the middle area of ​​the intercepted fused picture contains the middle area of ​​the display data of multiple layers, so that the captured target image stream is adapted to the resolution of the virtual screen, thereby improving the accuracy of picture capture on the main screen, and then improving the display quality of the target image on the virtual screen.

[0208] Among them, the texture coordinate scaling process of the fusion image is the same as Figure 4 The process of generating the target image stream is similar to that in

[15] and will not be described here.

[0209] In the embodiment of the present application, regardless of whether multiple layers are displayed on the screen together or separately, a fused picture of multiple layers can be obtained, and different display data on multiple layers can be displayed simultaneously, thereby ensuring that the fused picture is consistent with the picture displayed on the monocular screen. In this way, when the main screen is subsequently captured, the display data of multiple layers can be captured from the fused picture, thereby improving the accuracy of capturing the 3D application picture on the main screen, and thereby improving the display quality of the target image on the virtual screen.

[0210] See also Figure 11 , is a screen image sharing method provided in an embodiment of the present application, which is executed by a head-mounted display device and mainly includes the following steps:

[0211] S1101: Start a 3D application.

[0212] In some embodiments, when the head-mounted display device displays a 3D application, the display data of the 3D application is simultaneously displayed on the binocular display screens corresponding to the left and right eyes.

[0213] S1102: In response to a sharing instruction for the main screen, obtaining an initial image stream of the main screen.

[0214] Generally, the images of 3D applications on the binocular display screens corresponding to the left and right eyes are different but the difference is small. In most cases, there is no need to share the 3D images of the binocular display screens. Therefore, in order to save device memory and improve sharing efficiency, the display data of any one of the binocular display screens (recorded as the main screen) can be shared.

[0215] In some embodiments, after the head-mounted display device receives a user-triggered instruction to share the main screen, it obtains an initial image stream of the main screen, wherein each initial image frame in the initial image stream contains display data of the 3D application.

[0216] In some embodiments, the capture instruction for the main screen includes but is not limited to a screen casting operation.

[0217] S1103: Obtaining width and height information of the display screen of the external device according to the sharing instruction.

[0218] Among them, external devices include but are not limited to 2D display terminals such as smart TVs, smart phones, laptops, desktop computers, car terminals, projectors, etc. The display screen of the external device is used to display the display data of the 3D application shared from the main screen.

[0219] Specifically, when performing sharing operations such as casting the main screen, the head-mounted display device is connected to an external device via Miracast or HDMI, so that the display screen of the external device is used as a secondary screen to display the cast image on the main screen.

[0220] Since the main screen is generally square or approximately square, and the display screen of the external device is generally rectangular, in order to ensure normal 2D display of the initial image stream on the main screen, the width and height information of the display screen of the external device may be obtained.

[0221] It should be noted that the method for obtaining the width and height information of the display screen of the external device is similar to the method for obtaining the width and height information of the virtual screen, and will not be repeated here.

[0222] S1104: For each initial image frame in the initial image stream, scale the texture coordinates of the display data of the 3D application to the aspect ratio determined by the aspect information to capture the middle area of ​​the display data to generate a target image stream.

[0223] In some embodiments, a class inheriting from Presentation is created to create and manage the viewport displayed by the external device. The SurfaceHolder.Callback interface implemented in this class can handle events such as the creation, modification, and destruction of Surface objects. This allows for timely notification and execution of corresponding actions when the Surface object of the external device display is ready.

[0224] Since the aspect ratio of the main screen is 1:1 or approximately 1:1, the main screen of the head-mounted display device is square or approximately square, while the display screen of the external device is mostly rectangular. In order to enable the display data of the 3D application on the main screen to be normally rendered on the display screen of the external device, the texture coordinates of the display data of the 3D application on the main screen can be scaled according to the width and height information of the display screen of the external device. The aspect ratio of the scaled texture coordinates is the same as the aspect ratio of the display screen of the external device, thereby cutting out the middle area of ​​the display data of the 3D application on the main screen, so that the target image stream containing the cut-out middle area is adapted to the resolution of the display screen of the external device.

[0225] It should be noted that the method of generating the target image stream during the main screen sharing operation is similar to the method of generating the target image stream during the main screen capture operation. For details, see Figure 4 , I will not go into details here.

[0226] In an embodiment of the present application, the texture coordinate scaling ratio is calculated using the width and height information of the external device display screen, and the texture coordinates of the vertices of two triangles are generated based on the texture coordinate scaling ratio. In this way, when the display data of the 3D application on the main screen is sampled based on the texture coordinates, the intercepted target image can be adapted to the size of the external device display screen, thereby avoiding operations such as filling and stretching the target image, and improving the quality of the shared target image. In addition, the method for calculating the texture coordinate scaling ratio can achieve the interception of the middle area of ​​the display data of the 3D application on the main screen, thereby ensuring that the intercepted target image retains more areas of user interest. Since the value range of the texture coordinates is [0,1], and considering that the current display of 2D images is generally rectangular, two triangles with a value range of [0,1] and that can form a rectangle are generated based on the texture scaling coordinate ratio, thereby ensuring the normal display of the target image on the display screen of the external device.

[0227] When sharing the display data of a 3D application on the main screen, if the display of the 3D application occupies multiple layers and different layers correspond to different display data of the 3D application, each frame of the initial image stream obtained includes the display data of multiple layers. In this case, for the sharing process of the main screen, the method for generating the target image stream in S1104 is as follows: Figure 12 , mainly includes the following steps:

[0228] S1104_1: Obtain a fusion image of display data of the 3D application on multiple layers in the initial image according to a screen-on-screen mode of the multiple layers.

[0229] Because multiple layers are displayed differently, the results of sharing the main screen are also different. To prevent the display method from affecting the sharing results, a second thread is called to share the main screen. Through inter-thread communication and data transfer, the display data of multiple layers on the main screen can be passed to the second thread, and the surface object of the external device's display is also passed to the second thread. In this way, the content captured on the main screen is rendered on the external device's display through the second thread, ensuring that the display content of the external device and the main screen is synchronized.

[0230] Specifically, the data acquisition process for different screen-up methods is as follows:

[0231] S1104_11: Determine the screen display method for multiple layers. If they are displayed together, execute S1104_12; if they are displayed separately, execute S1104_13.

[0232] S1104_12: Use the second thread to directly obtain the fusion image generated by the first thread according to the display data on multiple layers.

[0233] Among them, the first thread is the rendering thread of the 3D application, and the second thread is the thread corresponding to the screen operation. Since the first thread has fused the display data of multiple layers when rendering the display data of multiple layers to the main screen, at this time, the second thread can directly obtain the fused screen of the display data of multiple layers generated by the first thread for subsequent operations.

[0234] S1104_13: Use the second thread to obtain display data on each layer from the first thread, and fuse the display data to generate a fused image.

[0235] Since the first thread renders the display data of multiple layers onto the main screen separately, at this time, in order to ensure that the display screen of the external device and the display content of the main screen are synchronized, the second thread can be used to obtain the display data of each layer from the first thread separately, and the various display data are fused to generate a fused screen for subsequent operations.

[0236] It should be noted that the fusion process of the display data of each layer when sharing the display data of the 3D application on the main screen is similar to the fusion process and rendering process of the display data of each layer when capturing the display data of the 3D application on the main screen, and will not be repeated here.

[0237] S1104_2: Scale the texture coordinates of the fused image to the aspect ratio determined by the width and height information, so as to capture the middle area of ​​the display data to generate a target image stream.

[0238] When the display of the 3D application on the main screen occupies multiple layers, and different layers are used to display different display data, no matter multiple layers are displayed on the screen together or separately, in order to fully display all the content on the main screen, it is necessary to obtain a fused picture of the display data on multiple layers. The fused picture contains all the display data of multiple layers on the main screen. In this way, when the texture coordinates of the fused picture are subsequently scaled according to the width and height information of the display screen of the external device, it can be ensured that the middle area of ​​the intercepted fused picture contains the middle area of ​​the display data of multiple layers, so that the captured target image stream is adapted to the resolution of the display screen of the external device, thereby improving the quality of the target image displayed on the display screen of the external device.

[0239] Among them, the texture coordinate scaling process of the fusion image is the same as Figure 4 The process of generating the target image stream is similar to that in

[15] and will not be described here.

[0240] S1105: Send the target image stream to the external device, so that the external device renders the target image stream on the display screen.

[0241] It should be noted that the rendering process of the target image stream by the external device is relatively mature and belongs to the existing technology, and will not be described in detail here.

[0242] In an embodiment of the present application, the middle area of ​​the 3D application screen on the main screen is captured and shared through the width and height information of the display screen of the external device, so that the resolution of the shared target image stream is adapted to the resolution of the display screen of the external device, thereby achieving the normal display of the shared screen on the main screen of the head-mounted display device on the display screen of the external device, avoiding problems such as screen filling and stretching, and improving the quality of screen sharing.

[0243] Take the example of projecting the 3D image of the main screen to the smart TV. Figure 13A As shown, the display screen of the smart TV is used as a secondary screen to display the central area image of the 3D application screen on the main screen of the head-mounted display device, realizing screen sharing of the monocular image without distortion and effectively solving the problem of black edges on both sides or picture stretching.

[0244] like Figure 13B As shown, when the user wears a head-mounted display device for a 3D experience and turns on the perspective function, two layers are superimposed on the main screen, one layer is used to display the real scene image captured by the camera, and the other layer is used to display the 3D application screen. When the picture displayed on the main screen is projected onto the smart TV, the smart TV can display the middle area image after the two layers are fused, and the displayed content is consistent with the content displayed on the main screen.

[0245] Based on the same technical concept, an embodiment of the present application provides a head-mounted display device that can implement the steps of the above-mentioned screen image capturing method and sharing method, and can achieve the same technical effect.

[0246] See also Figure 14 The head-mounted display device includes a processor 1401, a memory 1402, a binocular display screen 1403 and a communication interface 1404, wherein the communication interface 1404, the binocular display screen 1403, the memory 1402 and the processor 1401 are connected via a bus 1405;

[0247] The communication interface 1404 is used to communicate with external devices;

[0248] The binocular display screen 1403 is used to display images corresponding to the left and right eyes;

[0249] The memory 1402 stores a computer program, and the processor 1401 executes the steps of any one of the screen capture methods or screen sharing methods according to the computer program.

[0250] In an embodiment of the present application, the memory 1402 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and programs required to run the instant messaging function, etc.; the data storage area may store various instant messaging information and operating instruction sets, etc. The memory 1402 may be a volatile memory (volatile memory), such as a random-access memory (RAM); the memory 1402 may also be a non-volatile memory (non-volatile memory), such as a read-only memory, a flash memory (flash memory), a hard disk drive (HDD) or a solid-state drive (SSD); or the memory 1402 may be any other medium that can be used to carry or store a desired computer program in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 1402 may be a combination of the above memories.

[0251] The processor 1401 may include one or more central processing units (CPUs), GPUs, or digital processing units, etc. The processor 1401 is configured to implement the steps of any of the above-mentioned screen capture methods or screen sharing methods when calling a computer program stored in the memory 1402 .

[0252] It should be noted that Figure 14 This is merely an example of the hardware necessary for a head-mounted display device to execute the steps of the screen capture method or screen sharing method provided in the embodiments of this application. Not shown, the head-mounted display device may also include conventional hardware such as a sound pickup, a microphone, an IMU, a camera, a power supply, and a controller.

[0253] In the embodiment of the present application, the specific connection medium between the communication interface 1404, the binocular display screen 1403, the memory 1402 and the processor 1401 is not limited. In the embodiment of the present application, the communication interface 1404 and the binocular display screen 1403 are connected to the bus 1405 between the memory 1402 and the processor 1401. Figure 14 The connections between the other components are shown in bold lines for illustration only and are not intended to be limiting. The bus 1405 can be divided into an address bus, a data bus, a control bus, etc. For ease of description, Figure 14 The diagram shows a single thick line, but this does not indicate that there is only one bus or one type of bus.

[0254] For the convenience of description, the head mounted display device can be divided into modules (or units) according to their functions and described separately. Of course, when implementing this application, the functions of each module (or unit) can be implemented in the same or multiple software or hardware.

[0255] Those skilled in the art will appreciate that various aspects of the present application can be implemented as systems, methods, or program products. Therefore, various aspects of the present application can be specifically implemented in the following forms: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or an implementation that combines hardware and software aspects, which may be collectively referred to herein as a "circuit," "module," or "system."

[0256] An embodiment of the present application further provides a computer-readable storage medium for storing some instructions. When these instructions are executed, the steps of any one of the screen capture methods or screen sharing methods in the aforementioned embodiments can be completed.

[0257] An embodiment of the present application further provides a computer program product for storing a computer program, wherein the computer program is used to execute the steps of any one of the screen capture methods or screen sharing methods in the aforementioned embodiments.

[0258] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0259] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0260] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device that implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0261] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0262] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. A method for capturing a screen image, characterized in that: Applied to a head-mounted display device, the method includes: Start the 3D application; In response to a capture instruction for a main screen, acquiring an initial image stream of the main screen, where the main screen is any one of the binocular display screens, the main screen is used to display the 3D application, and each initial image frame in the initial image stream includes display data of the 3D application; generating a virtual screen of the initial image stream according to the capture instruction, and obtaining preset width and height information of the virtual screen; For each initial image frame in the initial image stream, scaling the texture coordinates of the display data of the 3D application to an aspect ratio determined by the aspect information, so as to intercept a middle area of ​​the display data to generate a target image stream; The target image stream is rendered onto the virtual screen.

2. The method according to claim 1, wherein When the display of the 3D application occupies multiple layers, and different layers correspond to different display data of the 3D application, scaling the texture coordinates of the display data of the 3D application to a ratio equal to the aspect ratio determined by the aspect-height information to intercept a middle area of ​​the display data to generate a target image stream includes: Obtaining, according to the on-screen manner of the multiple layers, a fused image of the display data of the 3D application on the multiple layers in the initial image; The texture coordinates of the fused image are scaled to an aspect ratio determined by the aspect information, so as to intercept a middle area of ​​the fused image to generate a target image stream.

3. The method according to claim 2, wherein The step of obtaining a fused image of display data of the 3D application on the multiple layers in the initial image according to the on-screen manner of the multiple layers includes: When the on-screen display mode is to display the multiple layers on the screen together, a second thread is used to directly obtain a fused image generated by the first thread based on the display data on the multiple layers; wherein the first thread is a rendering thread of the 3D application, and the second thread is a thread corresponding to the screen operation; When the on-screen display mode is to display the multiple layers on the screen separately, the second thread is used to obtain the display data on each layer from the first thread, and the display data are merged to generate a merged image.

4. The method according to claim 3, wherein The adopting the second thread to respectively obtain the display data on each layer and fusing the display data to generate a fused image includes: Creating a target texture object, where the target texture object is used to store the fused image; Binding the target texture object to the target texture unit of the shader, and setting parameters of the target texture object; Using a frame buffer object, the source texture of the display data on each layer is rendered to the target texture object in the shader to generate a fused image, and the frame buffer object is used for off-screen rendering.

5. The method according to any one of claims 1 to 4, wherein The target image stream is generated in the following manner: Calculating a texture coordinate scaling ratio according to the width and height information, wherein a value range of the texture coordinate scaling ratio is (0, 1); Generating texture coordinates of vertices of two triangles according to the texture coordinate scaling ratio; wherein the two triangles form a rectangle, and the value range of the texture coordinates is [0, 1]; A first picture is rendered according to the texture coordinates of the vertices of the two triangles to generate a target image stream, wherein the first picture is the original display data of the 3D application or a fusion picture of the display data of the 3D application on multiple layers.

6. The method according to claim 5, wherein The formula for the texture coordinate scaling ratio is expressed as: scale=(1.0fH / W) / 2.0f The texture coordinates of the vertices of the two triangles are respectively expressed as: (0.0, scale), (0.0, 1.0f-scale), (1.0, 1.0-scale), (0.0, scale), (1.0, 1.0f-scale), (1.0, scale); Wherein, f represents a floating point number, H represents the height of the secondary screen, and W represents the width of the secondary screen.

7. The method according to any one of claims 1 to 4, wherein Rendering the target image stream onto the virtual screen includes: Setting the viewport size according to the preset width and height information of the virtual screen, and setting a rendering cropping area of ​​the same size according to the viewport size; For the current target image in the target image stream, clearing the color buffer of the previous target image and clearing the depth buffer of the previous target image according to a preset clear screen color; Calling the target rendering program of the shader, obtaining the vertex arrays corresponding to the two triangles generated based on the width and height information and setting the pointer of the vertex array; activating a target texture unit and binding the texture of the current target image to the target texture unit; Setting the texture parameters of the current target image; Binding the target texture unit to the sampler of the shader; Using the pointer to read the vertex array to draw the geometry in the rendering clipping area; The geometry is rendered in the sampler according to the target texture unit to render the current target image onto the virtual screen.

8. A method for sharing a screen image, characterized in that: Applied to a head-mounted display device, the method comprises: Start the 3D application; In response to a sharing instruction for a main screen, obtaining an initial image stream of the main screen, where the main screen is any one of the binocular display screens, the main screen is used to display the 3D application, and each initial image frame in the initial image stream includes display data of the 3D application; According to the sharing instruction, obtaining the width and height information of the display screen of the external device; For each initial image frame in the initial image stream, scaling the texture coordinates of the display data of the 3D application to a ratio that is the same as the aspect ratio determined by the aspect information, so as to intercept a middle area of ​​the display data to generate a target image stream; The target image stream is sent to the external device, so that the external device renders the target image stream on the display screen.

9. The method according to claim 8, wherein When the display of the 3D application occupies multiple layers, and different layers correspond to different display data of the 3D application, scaling the texture coordinates of the display data of the 3D application to a ratio equal to the aspect ratio determined by the aspect-height information to intercept a middle area of ​​the display data to generate a target image stream includes: Obtaining, according to the on-screen manner of the multiple layers, a fused image of the display data of the 3D application on the multiple layers in the initial image; The texture coordinates of the fused image are scaled to a ratio that is the same as the aspect ratio determined by the aspect information, so as to intercept a middle area of ​​the fused image to generate a target image stream.

10. A head-mounted display device, characterized in that: It includes a processor, a memory, a binocular display screen and a communication interface, wherein the communication interface, the binocular display screen, the memory and the processor are connected via a bus; The communication interface is used to communicate with external devices; The binocular display screen is used to display images corresponding to the left and right eyes; The memory stores a computer program, and the processor executes the method according to any one of claims 1 to 9.

Images