Rendering picture screen recording method, head-mounted display device, readable medium and product

Abstract

The embodiment of the invention discloses a rendering picture screen recording method, head-mounted display equipment, a readable medium and a product. A specific embodiment of the method comprises the steps of obtaining pose information, a target to-be-rendered picture frame and shared texture data of the head-mounted display device; determining distortion correction information of the target to-be-rendered picture frame according to the pose information; in response to the detection that the historical augmented reality screen recording video shakes, performing time dimension smoothing processing on the pose information to obtain smoothed pose information; and according to the distortion correction information, the post-smoothing pose information and the shared texture data, performing overall picture rendering on the target to-be-rendered picture frame to obtain a rendered picture frame, so that the server performs screen recording processing on the historical augmented reality screen recording video and the rendered picture frame to obtain an augmented reality screen recording video. According to the embodiment, the problems of picture tearing and snowflake points of the recorded video can be avoided, the time delay of video recording is further reduced, and the problem of video picture jitter during recording is avoided.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, and in particular to a rendering screen recording method, a head-mounted display device, a readable medium, and a product. Background Art

[0002] The screen recording technology of head-mounted display devices is a technology that projects the superimposed virtual screen and real screen onto an external screen to record it as a video. For the projection and recording of rendered screens, the usual method is: the application process independently renders the screen to the screen through a single-channel stereo rendering algorithm (SinglePass) and an automatic refresh rendering algorithm (AutoRefesh) to obtain a rendered screen frame set. Then, the server grabs the rendered screen frame set through a virtual screen display method (SurfaceFlinger), renders the virtual screen screen onto a canvas (Surface), and encodes and records the rendered screen frame set.

[0003] However, it is found in practice that when the above method is used to project and record the rendered screen, the following technical problems often occur: due to the use of a single-channel stereo rendering algorithm and an automatic refresh rendering algorithm, the server cannot accurately obtain the rendering status of the screen, resulting in tearing or snowflakes in the recorded screen, reducing the quality of the recorded screen, and the screen rendering is based on the three-degree-of-freedom posture information. The head-mounted display device is very sensitive, and the slightest movement of the human body will be fed back to the rendered screen, resulting in frequent jitter in the recorded screen, reducing the quality of the recorded screen. Summary of the invention

[0004] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0005] Some embodiments of the present disclosure propose a rendering screen recording method, a head-mounted display device, a readable medium and a product to solve one or more of the technical problems mentioned in the above background technology section.

[0006] In a first aspect, some embodiments of the present disclosure provide a method for recording a rendered screen, comprising: obtaining posture information of a head-mounted display device, a target screen frame to be rendered, and shared texture data of a target application process; determining, based on the above posture information, the distortion correction information of the above target screen frame to be rendered; in response to detecting jitter in a historical augmented reality screen recording video, performing time dimension smoothing on the above posture information to obtain smoothed posture information, wherein the above historical augmented reality screen recording video is a screen recording video corresponding to each screen frame to be rendered that is located before the above target screen frame to be rendered; based on the above distortion correction information, the above smoothed posture information and the above shared texture data, performing overall rendering on the above target screen frame to be rendered to obtain a rendered screen frame, so that the server can perform screen recording processing on the above historical augmented reality screen recording video and the above rendered screen frame to obtain an augmented reality screen recording video.

[0007] Optionally, the method also includes: in response to detecting that the rendering of the above-mentioned target application process is completed and the current application process meets the conditions for starting the rendering operation, obtaining the shared texture data corresponding to the above-mentioned current application process, the posture information of the head-mounted display device and the target picture frame to be rendered, so as to perform overall rendering of the picture.

[0008] Optionally, the above-mentioned determining the distortion correction information of the target to-be-rendered screen frame based on the above-mentioned posture information includes: obtaining the device calibration information and screen display position information of the above-mentioned head-mounted display device, wherein the above-mentioned screen display position information is position information extracted from the posture information at a previous time point; determining a first position difference between the above-mentioned screen display position information and the above-mentioned posture information; determining the target mapping display position information of the above-mentioned target to-be-rendered screen frame based on the above-mentioned first position difference; performing anti-distortion fusion processing on the above-mentioned device calibration information and the above-mentioned first position difference to obtain a second position difference; and determining the above-mentioned target mapping display position information and the above-mentioned second position difference as the distortion correction information.

[0009] Optionally, the above-mentioned posture information is smoothed in the time dimension to obtain smoothed posture information, including: determining the average posture information of the above-mentioned target to-be-rendered screen frame and the previous to-be-rendered screen frame; determining the screen rotation angle according to the above-mentioned average posture information; determining the product of the derivative of the above-mentioned screen rotation angle with respect to time and the angular velocity in the above-mentioned posture information as the screen offset; determining the screen smoothness of the above-mentioned target to-be-rendered screen frame according to the above-mentioned screen offset; generating the smoothed posture information according to the above-mentioned screen smoothness and the above-mentioned average posture information.

[0010] Optionally, the method also includes: controlling the low-latency rendering thread included in the above-mentioned target application process, and performing low-latency parallel rendering on the above-mentioned target to-be-rendered screen frames according to the above-mentioned shared texture data to obtain low-latency rendered screen frames; determining each obtained low-latency rendered screen frame as a projection video, wherein the above-mentioned each low-latency rendered screen frame is each screen frame obtained by low-latency rendering.

[0011] Optionally, the above-mentioned acquisition of posture information of the head-mounted display device, the target screen frame to be rendered and the shared texture data of the target application process includes: controlling the low-latency rendering thread and the screen recording rendering thread included in the above-mentioned target application process to share an open image library context, wherein the above-mentioned open image library context is the rendering state and resource information required for rendering created by the above-mentioned low-latency rendering thread or the above-mentioned screen recording rendering thread; acquiring shared texture data through the above-mentioned open image library context; acquiring posture information through the above-mentioned head-mounted display device; and acquiring the target screen frame to be rendered corresponding to the above-mentioned target application process.

[0012] Optionally, the above-mentioned augmented reality screen recording video is obtained through the following steps: in response to determining to communicate with the above-mentioned target application process through the image canvas communication mechanism, obtaining the initial rendering screen frame corresponding to the above-mentioned target application process, and starting the recording operation, wherein the above-mentioned initial rendering screen frame is the first rendering screen frame in the above-mentioned historical augmented reality screen recording video; in response to detecting that the recording operation is successfully started, the above-mentioned historical augmented reality screen recording video and the above-mentioned rendering screen frame are video encoded to obtain an augmented reality screen recording video bitstream as the augmented reality screen recording video.

[0013] In a second aspect, some embodiments of the present disclosure provide a head-mounted display device, comprising: one or more processors; a storage device for storing one or more programs; at least one display screen and an optical element for forming an image in front of a user's eyes; when the one or more programs are executed by one or more processors, the one or more processors implement the method described in any implementation manner of the above-mentioned first aspect.

[0014] In a third aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method described in any implementation manner in the first aspect is implemented.

[0015] In a fourth aspect, some embodiments of the present disclosure provide a computer program product, including a computer program, which implements the method described in any implementation manner of the above-mentioned first aspect when executed by a processor.

[0016] The above-mentioned embodiments of the present disclosure have the following beneficial effects: the screen-dyeing and screen-recording methods of some embodiments of the present disclosure can avoid the problems of screen tearing and snowflakes in the recorded video, further reduce the delay of the recorded video, and avoid the problem of video screen jitter during recording. Specifically, the relevant recorded screen is torn or snowflakes, which reduces the quality of the recorded screen, and the screen-recording screen has the problem of frequent jitters. The reason for reducing the quality of the recorded screen is that due to the use of a single-channel stereo rendering algorithm and an automatic refresh rendering algorithm, the server cannot accurately obtain the rendering status of the screen rendering, resulting in tearing or snowflakes in the recorded screen, reducing the quality of the recorded screen, and the screen rendering is based on the rendering of the three-degree-of-freedom posture information. The sensitivity of the head-mounted display device is very high, and the tiny movements of the human body will be fed back to the rendering screen, resulting in the problem of frequent jitters in the recorded screen, reducing the quality of the recorded screen. Based on this, the screen-dyeing and screen-recording methods of some embodiments of the present disclosure can first obtain the posture information of the head-mounted display device, the target screen frame to be rendered, and the shared texture data of the target application process. Thus, the obtained posture information of the head-mounted display device, the target to-be-rendered screen frame and the shared texture data of the target application process can be used for the subsequent rendering of the target to-be-rendered screen frame. Secondly, according to the above-mentioned posture information, the distortion correction information of the above-mentioned target to-be-rendered screen frame is determined. Thus, the distortion correction information can remove the information that the virtual screen in the head-mounted display device is distorted, and the modification of the posture information can reduce the rendering delay and the subsequent distortion of the video recorded by the server, and improve the picture quality. Then, in response to the detection of the jitter of the historical augmented reality screen recording video, the above-mentioned posture information is smoothed in the time dimension to obtain the smoothed posture information, wherein the above-mentioned historical augmented reality screen recording video is the screen recording video corresponding to each to-be-rendered screen frame located before the above-mentioned target to-be-rendered screen frame. Therefore, due to the sensitivity of the head-mounted display device, slight shaking of the human body will be fed back to the virtual reality screen to cause the jitter of the screen, and the time dimension smoothing can offset the screen jitter caused by slight shaking of the human body, avoiding the problem of anti-shake of the recorded video screen. Finally, based on the distortion correction information, the smoothed pose information and the shared texture data, the target to-be-rendered screen frame is rendered as a whole to obtain a rendered screen frame, so that the server can record the historical augmented reality screen video and the rendered screen frame to obtain an augmented reality screen video. Therefore, by rendering the target to-be-rendered screen as a whole, the problems of screen tearing and snowflakes can be avoided, the quality of the recorded video can be improved, and the rendering delay can be further reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0018] Figure 1 is a schematic diagram of an application scenario of a rendering screen recording method according to some embodiments of the present disclosure;

[0019] Figure 2 is a flowchart of some embodiments of the rendering screen recording method according to the present disclosure;

[0020] Figure 3 is a flowchart of other embodiments of the rendering screen recording method according to the present disclosure;

[0021] Figure 4 It is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0022] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0023] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0024] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0025] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0026] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0027] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0028] Figure 1 It is a schematic diagram of an application scenario of the rendering screen recording method according to some embodiments of the present disclosure.

[0029] like Figure 1 As shown, first, the posture information of the head-mounted display device 101, the target screen frame to be rendered 102 and the shared texture data of the target application process are obtained. According to the above shared texture data and the above posture information, the distortion correction information 103 of the above target screen frame to be rendered is determined. Then, in response to the detection of jitter in the historical augmented reality screen recording video, the above posture information is smoothed in the time dimension to obtain the smoothed posture information 104, wherein the above historical augmented reality screen recording video is the screen recording video corresponding to each screen frame to be rendered located before the above target screen frame to be rendered. Finally, according to the above distortion correction information and the above smoothed posture information, the above target screen frame to be rendered is rendered to obtain a rendered screen frame, so that the server 107 can perform screen recording processing on the above historical augmented reality screen recording video and the above rendering screen frame to obtain an augmented reality screen recording video 108. In this application scenario, the connection with the server can be connected through the network 106.

[0030] It is understandable that the execution subject of the rendering screen recording method can be a processor of a head-mounted display device, or it can be an associated terminal device. Among them, the head-mounted display device can be an AR head-mounted display device, a VR head-mounted display device, or an MR head-mounted display device. When the execution subject of the rendering screen recording method is software, it can be installed in the head-mounted display device listed above. It can be implemented as multiple software or software modules for providing distributed services, for example, or it can be implemented as a single software or software module. No specific limitation is made here.

[0031] Continue to refer Figure 2 , shows a process 200 of some embodiments of the rendering screen recording method according to the present disclosure. The rendering screen recording method comprises the following steps:

[0032] Step 201, obtaining the posture information of the head mounted display device, the target picture frame to be rendered, and the shared texture data of the target application process.

[0033] In some embodiments, the execution subject of the above-mentioned rendering screen recording method (such as a head-mounted display device) can obtain the posture information of the head-mounted display device, the target to-be-rendered screen frame and the shared texture data of the target application process through a wired connection or a wireless connection. Among them, the above-mentioned posture information can be the angle and positioning position information of the target user wearing the above-mentioned head-mounted display device rotating around the three coordinate axes of the horizontal axis, the vertical axis and the longitudinal axis. The above-mentioned target to-be-rendered screen frame can be a frame of image currently waiting to be rendered. The above-mentioned target application process can be an application process installed in the above-mentioned head-mounted display device and waiting to render the screen. The above-mentioned shared texture data can be image data used to color the above-mentioned target to-be-rendered screen frame and provide screen details. It should be pointed out that the above-mentioned wireless connection method can include but is not limited to 3G / 4G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0034] In some optional implementations of some embodiments, the execution subject may obtain the posture information of the head mounted display device, the target picture frame to be rendered, and the shared texture data of the target application process through the following steps:

[0035] The first step is to control the low-latency rendering thread and the screen recording rendering thread included in the target application process to share the open image library context, wherein the open image library context may be the rendering state and the resource information required for rendering created by the low-latency rendering thread or the screen recording rendering thread. Among them, the low-latency rendering thread may be a thread that renders the picture to the screen by reducing the delay of the picture. The screen recording rendering thread may be a rendering thread that records the rendered picture as a whole on the server side. The rendering state may be information that records the state of rendering-related operations in the OpenGL (Open Graphics Library) environment. The rendering-related operations may include but are not limited to at least one of the following: binding operations of texture units and texture objects, texture filtering operations, and transparent object rendering operations. The resource information required for the rendering may include but is not limited to at least one of the following: texture data, shaders, shader programs, and buffer objects. The shader may be a software program for processing vertex or fragment shading operations of objects. The buffer object may be a GPU (Graphics Processing Unit) memory area for storing vertex data, index data, and rendering results. It should be noted that the low-latency rendering thread and the screen recording rendering thread can simplify the rendering processing operations by using shared texture data for screen rendering, reduce the storage space required for rendering, and further reduce the rendering load.

[0036] The second step is to obtain the shared texture data through the open image library context. In practice, the execution subject can obtain the shared texture data from the Unity development platform software through the open image library context.

[0037] The third step is to obtain the posture information through the head mounted display device. In practice, the execution subject may obtain the posture information through a gyroscope sensor in the head mounted display device.

[0038] The fourth step is to obtain the target to-be-rendered screen frame corresponding to the target application process.

[0039] Step 202: Determine the distortion correction information of the target image frame to be rendered according to the posture information.

[0040] In some embodiments, the execution subject may determine the distortion correction information of the target frame to be rendered according to the posture information, wherein the distortion correction information may be the posture information obtained after the posture information is corrected.

[0041] As an example, the execution subject may determine the distortion correction information of the target frame to be rendered according to the posture information through a ray propagation path tracing algorithm and a wavelet analysis algorithm.

[0042] In some optional implementations of some embodiments, the execution subject may determine the distortion correction information of the target to-be-rendered picture frame according to the posture information through the following steps:

[0043] The first step is to obtain the device calibration information and screen display position information of the head mounted display device, wherein the screen display position information may be position information extracted from the posture information at the previous time point. The device calibration information may represent the tangent and rotation angle between the real screen and the augmented reality screen. The screen display position information may represent the two-dimensional coordinate position of the screen when it is rendered on the canvas (surface).

[0044] The second step is to determine a first position difference between the above-mentioned screen display position information and the above-mentioned posture information, wherein the above-mentioned first position difference represents the difference between the above-mentioned screen display position information and the above-mentioned posture information.

[0045] As an example, the execution entity may determine the product of the inverse matrix of the screen display position information and the position matrix corresponding to the posture information as the first position difference.

[0046] The third step is to determine the target mapping display position information of the target image frame to be rendered according to the first position difference, wherein the target display position information may represent the two-dimensional mapping coordinates of each coordinate point in the target image frame to be rendered.

[0047] As an example, the execution subject may first obtain the previous screen mapping display position information corresponding to the posture information at the previous time point and the projection matrix in the calibration of the head-mounted display device. The projection matrix represents the conversion relationship of converting the camera coordinate system to the screen coordinate system in the screen rendering. Then, the inverse matrix of the projection matrix, the first position difference and the matrix product of the previous screen mapping display position information are determined as the target mapping display position information.

[0048] The fourth step is to perform anti-distortion fusion processing on the device calibration information and the first position difference to obtain a second position difference, wherein the second position difference represents the position difference between the device calibration information and the first position difference.

[0049] As an example, the execution entity may determine the matrix product of the inverse matrix of the device calibration information and the first position difference to obtain the second position difference.

[0050] In the fifth step, the target mapping display position information and the second position difference are determined as distortion correction information.

[0051] Step 203, in response to detecting that the historical augmented reality screen recording video is jittery, the posture information is smoothed in the time dimension to obtain smoothed posture information.

[0052] In some embodiments, the execution subject may smooth the posture information in the time dimension in response to detecting jitter in the historical augmented reality screen recording video to obtain smoothed posture information, wherein the historical augmented reality screen recording video may be a screen recording video corresponding to each frame to be rendered that is located before the target frame to be rendered. The recorded video may be a video obtained by recording and encoding each frame to be rendered. The smoothed posture information may represent the posture information that is smoothed in the time dimension to avoid jitter in the recorded image due to slight jitter of the human body.

[0053] As an example, the execution subject may, in response to detecting that the historical augmented reality screen recording video is jittery, input the above-mentioned posture information into the posture time smoothing function to obtain the smoothed posture information. The above-mentioned posture time smoothing function may be X k , Y k , Z k It can represent the smoothed pose information, that is, the angles in the horizontal, vertical and vertical directions.k-1 , Y k-1 , Z k-1 It can represent the pose information, that is, the angles in the horizontal, vertical and vertical directions before time dimension smoothing. XD M , YD M , ZD M It can represent the angles detected by the temporal gesture sensor in the head mounted display device on the horizontal axis, vertical axis and vertical axis. M It can represent the time when the attitude sensor uploads the detected angle. k It can represent the rendering time of the target frame to be rendered. k-1 It can represent the rendering time of the last frame to be rendered. c can represent a constant value in the range of [0, 1].

[0054] In some optional implementations of some embodiments, the execution subject may perform time dimension smoothing on the posture information to obtain smoothed posture information through the following steps:

[0055] The first step is to determine the average pose information of the target frame to be rendered and the previous frame to be rendered. The previous frame to be rendered may be a frame that is located before the target frame to be rendered and has not been rendered. The average pose information may represent the average value of the pose information between the target frame to be rendered and the previous frame to be rendered in the time dimension.

[0056] As an example, the execution subject may input the screen coordinates corresponding to the target frame to be rendered and the screen coordinates corresponding to the previous frame to be rendered into the screen coordinate mean constraint function to obtain the average pose information of the target frame to be rendered and the previous frame to be rendered. The screen coordinate mean constraint function may be Wherein, Average(p1, p2, t) may represent average pose information. p1 may represent the screen coordinates of the target frame to be rendered. p2 may represent the screen coordinates of the previous frame to be rendered. t may represent the time dimension. θ may represent the angle of rotation of the head mounted display device on the horizontal axis, the vertical axis and the vertical axis, i.e., the Euler angle.

[0057] The second step is to determine the screen rotation angle according to the average posture information, wherein the screen rotation angle may represent the rotation angle of the target screen frame to be rendered and the previous screen frame to be rendered corresponding to the user perspective of the target user.

[0058] As an example, the execution subject may first determine the sum of the main diagonals of the pose matrix of the average pose information as the pose matrix trace number. Then, determine the difference between the pose matrix trace number and 1 and the ratio to 2 as the matrix trace ratio. Finally, determine the arccosine function of the matrix trace ratio as the screen rotation angle.

[0059] The third step is to determine the product of the derivative of the screen rotation angle with respect to time and the angular velocity in the above-mentioned posture information as the screen offset. The above-mentioned screen offset can represent the change between the screen coordinates of the above-mentioned target screen frame to be rendered and the corresponding screen coordinates in the previous screen frame to be rendered. The above-mentioned angular velocity can represent the angular velocity of the above-mentioned head-mounted display device rotating between the time difference corresponding to the above-mentioned target screen frame to be rendered and the above-mentioned previous screen frame to be rendered.

[0060] The fourth step is to determine the screen smoothness of the target frame to be rendered according to the screen offset, wherein the screen smoothness can represent the smoothness and continuity of the change between the screen coordinates of the target frame to be rendered and the corresponding screen coordinates in the previous frame to be rendered.

[0061] As an example, the execution subject may first, in response to determining that the screen offset is less than a preset offset threshold, add 1 to the historical screen smoothness. The preset offset threshold may be a preset offset threshold. For example, the preset offset threshold may be 0.05. The historical screen smoothness may be the screen offset corresponding to the historical augmented reality screen recording video. Then, in response to determining that the screen translation is greater than the preset offset threshold, the historical screen smoothness is reduced by 1. Finally, in response to determining that the screen translation is equal to the preset offset threshold, the historical screen smoothness is determined as the screen smoothness.

[0062] The fifth step is to generate smoothed posture information based on the above-mentioned picture smoothness and the above-mentioned average posture information.

[0063] As an example, the execution subject may normalize the sum of the average pose information of the preset picture smoothness to obtain smoothed pose information.

[0064] Step 204, based on the distortion correction information, smoothed posture information and shared texture data, the target to-be-rendered screen frame is rendered as a whole to obtain a rendered screen frame, so that the server can record the historical augmented reality screen recording video and the rendered screen frame to obtain an augmented reality screen recording video.

[0065] In some embodiments, the execution subject may render the target to-be-rendered screen frame as a whole according to the distortion correction information, the smoothed posture information and the shared texture data to obtain a rendered screen frame, so that the server can record the historical augmented reality screen recording video and the rendered screen frame to obtain an augmented reality screen recording video. The rendered screen frame may be a screen frame after rendering is completed, which is presented in the head-mounted display screen and is a fusion of the virtual screen and the real screen. The augmented reality screen recording video may be a video of each rendered screen frame that is recorded by the server through a communication connection with the target application process.

[0066] As an example, the execution subject may map the shared texture data to the screen position corresponding to the distortion correction information and the smooth pose information to obtain a rendered screen frame. The server may use the SurfaceFlinger component to perform screen recording processing on the historical augmented reality screen recording video and the rendered screen frame to obtain an augmented reality screen recording video. The SurfaceFlinger component may be a component that synthesizes multiple layers into a display buffer and sends it to the canvas for display.

[0067] In some optional implementations of some embodiments, the augmented reality screen recording video may be obtained through the following steps:

[0068] The first step is to, in response to determining to establish a communication connection with the target application process through the image canvas communication mechanism, obtain the initial rendering screen frame corresponding to the target application process, and start the recording operation, wherein the initial rendering screen frame may be the first rendering screen frame in the historical augmented reality screen recording video. The image canvas communication (Surface Inter-Process Communication) mechanism may be a surface communication mechanism between the server and the application process running in the execution subject. The recording operation may be an operation acting on the server to record the rendered rendering screen.

[0069] In the second step, in response to detecting that the recording operation is successfully started, the above-mentioned historical augmented reality screen recording video and the above-mentioned rendering picture frame are video encoded to obtain an augmented reality screen recording video bit stream as the augmented reality screen recording video.

[0070] In the process of adopting technical solutions to solve the above technical problem 1, the following technical problem 2 is often accompanied: Since the historical augmented reality screen recording video is a three-dimensional video, including two-dimensional texture video images and depth maps of multiple perspectives, and since the depth map is different from the two-dimensional video frame, including a large number of flat areas and sharp edge areas, how to encode the depth image to reduce the video encoding bit stream and reduce the waste of video storage resources. For the above technical problem 2, the conventional solution is generally: using a quadtree-based coding unit division method to divide the depth map into coding units to obtain a depth coding unit set. Then, according to the depth coding unit set, the depth map is intra-encoded to obtain a depth encoding bit stream. However, the above conventional solution still has the following problems: a recursive method is used to divide the layers, and the sum of the rate-distortion cost of the coding unit is calculated to determine whether it is necessary to continue dividing. The depth map has sharp edge areas that require more coding units to be divided, resulting in low division efficiency, extended division time, and a large amount of computing resources. Increase the load of server-side video encoding. The inventors took into account the shortcomings of the above conventional solutions and combined with the three-dimensional video encoding advantages / technical status of the inventor's company, we decided to adopt the following solution:

[0071] In some optional implementations of some embodiments, the execution subject may perform video encoding on the historical augmented reality screen recording video and the rendered screen frame to obtain the augmented reality screen recording video through the following steps:

[0072] The first step, in response to determining that the above-mentioned rendering screen frame and the above-mentioned historically terminated augmented reality screen recording video frame are not located in the same rendering screen frame group, intra-frame encoding is performed on the two-dimensional texture video frame to obtain a two-dimensional video bit stream, wherein the above-mentioned rendering screen frame group may be a set of rendering screen frames corresponding to the same time and different shooting positions, and the above-mentioned historically terminated augmented reality screen recording video frame may be a video frame located at the termination position in the above-mentioned historical augmented reality screen recording video. Among them, the above-mentioned two-dimensional texture video frame may be a video frame that displays the appearance information of the object included in the rendering screen frame in a two-dimensional form. The above-mentioned rendering screen frame also includes: a depth map. The above-mentioned depth map may be an image that represents the distance between the camera and the object in grayscale pixels. The above-mentioned intra-frame coding may be encoded using intra-frame coding in the HEVC (High Efficiency Video Coding) algorithm.

[0073] In the second step, the depth map included in the above-mentioned rendering picture frame is divided into coding units to obtain a depth coding unit set. The above-mentioned depth coding unit set may include depth coding units of different sizes. For example, the above-mentioned depth coding unit set may include: a 64*64 depth coding unit, a 32*32 depth coding unit, a 16*16 depth coding unit, and an 8*8 depth coding unit.

[0074] In the third step, for each depth coding unit in the depth coding unit set, the following determination steps are performed:

[0075] Sub-step 1, input the above-mentioned depth coding unit into the first convolution pooling layer included in the coding unit division model to obtain a first depth coding feature map, wherein the above-mentioned first convolution pooling layer may include: a first convolution layer and a first maximum pooling layer, and the above-mentioned coding unit division model may also include: a second convolution pooling layer, a non-local self-attention mechanism layer, a third convolution pooling layer, a fourth convolution pooling layer, a spatial pyramid pooling network and a plurality of fully connected layers. Among them, the above-mentioned first convolution layer may be a convolution neural network layer with 64 convolution kernels, a convolution kernel of 3*3, and a step size of 1. The above-mentioned first maximum pooling layer may be a maximum pooling layer with a pooling kernel of 2*2 and a step size of 2. The above-mentioned coding unit division model may be a model for determining whether the input depth coding unit needs to be further divided. The above-mentioned non-local self-attention mechanism layer may be an attention mechanism layer for identifying sharp edge areas in the depth map to determine the weights of each pixel in the sharp edge area. The above-mentioned spatial pyramid pooling network may be a convolution neural network that generates a fixed-size output for input feature maps of different sizes. The second convolutional pooling layer, the third convolutional pooling layer and the fourth convolutional pooling layer may be network layers composed of the same convolutional layers and pooling layers, which have different input data from the first convolutional pooling layer.

[0076] Sub-step 2: input the first depth encoding feature map into the second convolutional pooling layer to obtain a second depth encoding feature map.

[0077] Sub-step 3, inputting the second depth encoding feature map into the non-local self-attention mechanism layer to obtain a third depth encoding feature map. The third depth encoding feature map can represent the feature information of the sharp edge area of ​​the depth map. The non-local self-attention mechanism layer can determine the different weights of each feature in the second depth encoding feature map by the following steps:

[0078] First, the second depth feature map is input into three 1*1 convolutional layers respectively to obtain the first convolutional feature map, the second convolutional feature map and the third convolutional feature map. Among them, the size of the second depth coding feature map can be c*w*h, then the size of the first convolutional feature map, the second convolutional feature map and the third convolutional feature map can all be c / 2*w*h. c represents the number of channels, w represents the width of the feature map, and h represents the length of the feature map. The first convolutional feature map and the second convolutional feature map can represent the similarity of pixels at two positions. The third convolutional feature map can represent the information of the texture features of any pixel in the depth map, so as to facilitate the weighted summation operation of the first convolutional feature map and the second convolutional feature map.

[0079] Secondly, the first convolution feature map and the third convolution feature map are resized to obtain a fourth convolution feature map and a fifth convolution feature map, wherein the sizes of the fourth convolution feature map and the fifth convolution feature map may be c / 2*w / 2*h / 2.

[0080] Again, the fourth convolution feature map, the second convolution feature map and the fifth convolution feature map are feature stretched to obtain a first convolution stretch feature map, a second convolution stretch feature map and a third convolution stretch feature map. The second convolution stretch feature map may be a one-dimensional feature map of c / 2*wh. The first convolution stretch feature map and the third convolution stretch feature map may be one-dimensional feature maps of c / 2*wh / 4.

[0081] Subsequently, the second convolution stretch feature map and the third convolution stretch feature map are transposed to obtain a first transposed feature map and a second transposed feature map. The first transposed feature map may be a feature map of wh*c / 2. The second transposed feature map may be a feature map of wh / 4*c / 2.

[0082] Next, determine the matrix multiplication of the first convolution stretch feature map and the first transposed feature map to obtain a first similarity feature map. The first similarity feature map can represent the similarity between any two pixels in the depth map. The first similarity feature map can be a feature map of wh*wh / 4*c / 2.

[0083] Afterwards, the matrix multiplication of the normalized first similarity feature map and the second transposed feature map is determined to obtain a second similarity feature map, wherein the second similarity feature map may be a wh*c / 2 feature map.

[0084] Then, the second similarity feature map is dimensionally stretched to obtain a dimensionally stretched feature map, wherein the dimensionally stretched feature map may be a feature map of w*h*c / 2.

[0085] Finally, the dimensionally stretched feature map is expanded and concatenated with the second depth-encoded feature map to obtain a third depth-encoded feature map.

[0086] Sub-step 4: input the third depth encoding feature map into the third convolutional pooling layer to obtain a fourth depth encoding feature map.

[0087] Sub-step 5: input the fourth depth encoding feature map into the fourth convolutional pooling layer to obtain a fifth depth encoding feature map.

[0088] Sub-step 6, respectively input the third depth coding feature map and the preset quantization parameter set corresponding to the third depth coding feature map, the fifth depth coding feature map and the preset quantization parameter set corresponding to the fifth depth coding feature map into the spatial pyramid pooling network to obtain a sixth depth coding feature map set. Among them, the preset quantization parameter in the preset quantization parameter set can be a pre-set value for quantizing the coefficients after discrete cosine transformation to reduce the video accuracy. The size of each sixth depth coding feature map included in the sixth depth coding feature map set is the same feature map.

[0089] Sub-step 7, splicing the sixth depth coding feature atlas and inputting it into a multi-layer fully connected layer to obtain a coding unit division result. The multi-layer fully connected layer may be a 3-layer fully connected layer. The coding unit division result may represent whether the depth coding unit is further divided.

[0090] Sub-step 8: In response to determining that the coding unit division result indicates that the division is completed, intra-coding the depth coding unit to obtain a depth coding bit stream.

[0091] In a fourth step, in response to determining that the coding unit division result indicates continued division, the depth coding unit is again divided into coding units to obtain a depth division coding unit set, and the depth division coding unit set is determined as a depth coding unit set to continue to perform the above determination step.

[0092] In the sixth step, the two-dimensional video bit stream and the obtained depth coding bit streams are determined as the rendering picture bit stream.

[0093] The fifth step, in response to determining that the above-mentioned rendering screen frame and the above-mentioned historical terminated augmented reality screen recording video frame are located in the same rendering screen frame group, according to the above-mentioned historical augmented reality screen recording video, the two-dimensional texture video frame and depth map included in the above-mentioned rendering screen frame are inter-frame encoded respectively to obtain a rendering screen bit stream.

[0094] In practice, the above-mentioned execution entity can utilize the disparity compensation prediction, inter-viewpoint motion prediction, and inter-viewpoint redundancy prediction in the 3D-HEVC (3D High Efficiency Video Coding) algorithm to perform inter-frame encoding on the remaining video frame group according to the above-mentioned initial rendered picture frame and the above-mentioned initial depth map to obtain the remaining two-dimensional video bit stream group and the remaining depth coding bit stream group.

[0095] In the seventh step, the bit stream corresponding to the above-mentioned rendering picture bit stream and the above-mentioned historical augmented reality screen recording video is determined as the augmented reality screen recording video bit stream as the augmented reality screen recording video.

[0096] The above-mentioned first to seventh steps and their related contents are an inventive point of an embodiment of the present disclosure, which solves the technical problem 2 mentioned in the background technology, "using a recursive method to divide layer by layer, and calculating the sum of the rate-distortion cost of the coding unit to determine whether to continue dividing, the depth map has a sharp edge area and needs to divide more coding units, resulting in low division efficiency, prolonged division time, requiring a large amount of computing resources, and increasing the load of server-side video encoding". The factors that lead to low division efficiency, prolonged division time, and require a large amount of computing resources, and increase the load of server-side video encoding are often as follows: using a recursive method to divide layer by layer, and calculating the sum of the rate-distortion cost of the coding unit to determine whether to continue dividing, and the depth map has a sharp edge area and needs to divide more coding units. If the above factors are solved, the effect of improving division efficiency, reducing division time, reducing waste of computing resources, and reducing the load of server-side video encoding can be achieved. In order to achieve this effect, the present disclosure first determines whether the rendered screen frame and the historically terminated augmented reality screen recording video frame are located in the same rendered screen frame group, so as to determine the encoding method of the rendered screen frame under different circumstances. Secondly, when located in different rendering picture frame groups, the HEVC encoding algorithm is used to perform video encoding on the two-dimensional texture video frame included in the rendering picture frame, which can reduce the encoding bit stream while ensuring the resolution of the two-dimensional texture video frame and reduce the occupancy of storage resources. Subsequently, the depth map is encoded and divided to obtain a depth coding unit set, which is convenient for subsequent determination of whether each divided depth coding unit continues to be divided. Then, for each depth coding unit in the depth coding unit set, the following determination steps are performed: the first step is to perform secondary feature extraction on the depth coding unit through the convolution layer and the pooling layer, so that the feature information of the depth coding unit can be fully learned. The second step is to input the depth coding feature map extracted by the convolution pooling layer into the non-local self-attention mechanism layer. For the depth map including a large number of flat areas and sharp edge areas, a larger weight value can be given to the sharp edge area, and a smaller weight value can be given to the flat area, so as to enhance the attention to important information, remove a large amount of redundant information, improve the recognition of the depth map, and thus improve the accuracy of the division of the coding unit of the depth map. The third step is to perform secondary convolution pooling on the third depth coding feature map again, so as to fully extract the detail information of the depth map. In the fourth step, the third depth encoding feature map, the fifth depth encoding feature map and their corresponding preset quantization parameter sets are input into the spatial pyramid pooling network. The inputs of different sizes are pooled to obtain outputs of the same size, which can solve the defects of inputs of different sizes and avoid the loss of information included in the feature map.In the fifth step, the concatenated sixth depth coding feature map is input into the multi-layer fully connected layer, and the depth coding unit representing the end of the division is intra-coded, which can improve the accuracy of the depth coding unit division, and then accurately end the continued division of useless coding units in advance, improve the division efficiency of the depth coding unit, and improve the coding efficiency of the depth map. Finally, when located in the same rendering picture frame group, the rendering picture frame is inter-coded, which can reduce the bit stream generated after encoding and reduce the occupancy of storage resources.

[0097] Optionally, after performing overall rendering of the target to-be-rendered screen frame according to the distortion correction information, the smoothed posture information and the shared texture data to obtain a rendered screen frame, so that the server can perform screen recording processing on the historical augmented reality screen recording video and the rendered screen frame to obtain the augmented reality screen recording video, the method may further include:

[0098] The first step is to control the low-latency rendering thread included in the target application process, and perform low-latency parallel rendering on the target to-be-rendered screen frame according to the shared texture data to obtain a low-latency rendered screen frame. The low-latency rendered screen frame can be a screen image obtained by reducing the time interval from the input of the target to-be-rendered screen frame to the completion of the rendering, and completing the rendering of the color and texture details of the target to-be-rendered screen. It should be noted that the low-latency rendering thread and the recording rendering thread are independent parallel rendering threads.

[0099] As an example, the execution subject may control the low-latency rendering thread included in the target application process, and use the low-latency rendering algorithm to perform low-latency parallel rendering on the target to-be-rendered screen frame according to the shared texture data to obtain a low-latency rendered screen frame. The low-latency rendering algorithm may be a Single Buffer rendering algorithm or a PTW (Positional TimeWarp) algorithm.

[0100] In the second step, each low-latency rendered picture frame obtained is determined as a projection video, wherein each low-latency rendered picture frame can be a picture frame obtained by low-latency rendering. The projection video can be a video obtained by projecting each low-latency rendered picture frame onto a real screen.

[0101] The above-mentioned embodiments of the present disclosure have the following beneficial effects: the screen-dyeing and screen-recording methods of some embodiments of the present disclosure can avoid the problems of screen tearing and snowflakes in the recorded video, further reduce the delay of the recorded video, and avoid the problem of video screen jitter during recording. Specifically, the relevant recorded screen is torn or snowflakes, which reduces the quality of the recorded screen, and the screen-recording screen has the problem of frequent jitters. The reason for reducing the quality of the recorded screen is that due to the use of a single-channel stereo rendering algorithm and an automatic refresh rendering algorithm, the server cannot accurately obtain the rendering status of the screen rendering, resulting in tearing or snowflakes in the recorded screen, reducing the quality of the recorded screen, and the screen rendering is based on the rendering of the three-degree-of-freedom posture information. The sensitivity of the head-mounted display device is very high, and the tiny movements of the human body will be fed back to the rendering screen, resulting in the problem of frequent jitters in the recorded screen, reducing the quality of the recorded screen. Based on this, the screen-dyeing and screen-recording methods of some embodiments of the present disclosure can first obtain the posture information of the head-mounted display device, the target screen frame to be rendered, and the shared texture data of the target application process. Thus, the obtained posture information of the head-mounted display device, the target to-be-rendered screen frame and the shared texture data of the target application process can be used for the subsequent rendering of the target to-be-rendered screen frame. Secondly, according to the above-mentioned posture information, the distortion correction information of the above-mentioned target to-be-rendered screen frame is determined. Thus, the distortion correction information can remove the information that the virtual screen in the head-mounted display device is distorted, and the modification of the posture information can reduce the rendering delay and the subsequent distortion of the video recorded by the server, and improve the picture quality. Then, in response to the detection of the jitter of the historical augmented reality screen recording video, the above-mentioned posture information is smoothed in the time dimension to obtain the smoothed posture information, wherein the above-mentioned historical augmented reality screen recording video is the screen recording video corresponding to each to-be-rendered screen frame located before the above-mentioned target to-be-rendered screen frame. Therefore, due to the sensitivity of the head-mounted display device, slight shaking of the human body will be fed back to the virtual reality screen to cause the jitter of the screen, and the time dimension smoothing can offset the screen jitter caused by slight shaking of the human body, avoiding the problem of anti-shake of the recorded video screen. Finally, based on the distortion correction information, the smoothed pose information and the shared texture data, the target to-be-rendered screen frame is rendered as a whole to obtain a rendered screen frame, so that the server can record the historical augmented reality screen video and the rendered screen frame to obtain an augmented reality screen video. Therefore, by rendering the target to-be-rendered screen as a whole, the problems of screen tearing and snowflakes can be avoided, the quality of the recorded video can be improved, and the rendering delay can be further reduced.

[0102] Further references Figure 3 , shows a process 300 of another embodiment of the rendering screen recording method according to the present disclosure. The rendering screen recording method comprises the following steps:

[0103] Step 301, obtaining the posture information of the head mounted display device, the target picture frame to be rendered, and the shared texture data of the target application process.

[0104] Step 302: Determine the distortion correction information of the target image frame to be rendered according to the posture information.

[0105] Step 303, in response to detecting that the historical augmented reality screen recording video is jittery, the posture information is smoothed in the time dimension to obtain smoothed posture information.

[0106] Step 304, based on the distortion correction information, smoothed posture information and shared texture data, the target to-be-rendered screen frame is rendered as a whole to obtain a rendered screen frame, so that the server can record the historical augmented reality screen recording video and the rendered screen frame to obtain an augmented reality screen recording video.

[0107] In some embodiments, the specific implementation of steps 301-304 and the technical effects thereof can be referred to in Figure 2 The steps 201-204 in the corresponding embodiment are not described in detail here.

[0108] Step 305, in response to detecting that the rendering of the target application process is completed and the current application process meets the conditions for starting the rendering operation, the shared texture data corresponding to the current application process, the posture information of the head-mounted display device and the target to-be-rendered picture frame are obtained to perform overall rendering of the picture.

[0109] In some embodiments, the execution subject may, in response to detecting that the rendering of the target application process is finished and the current application process meets the conditions for starting the rendering operation, obtain the shared texture data corresponding to the current application process, the posture information of the head mounted display device and the target to-be-rendered screen frame to perform overall screen rendering. The current application process may be an application process different from the target application process that needs to perform screen rendering.

[0110] from Figure 3 It can be seen that Figure 2 Compared with the description of some corresponding embodiments, Figure 3 The corresponding process 300 of the rendering screen recording method in some embodiments embodies the steps of continuously recording the rendering screen between the target application process and the current application process. Therefore, the schemes described in these embodiments can record the continuous rendering screen when different application processes continuously exit and start the rendering operation, so as to achieve seamless recording between multiple application processes, thereby improving the recording fluency and the quality of the recorded rendering screen.

[0111] Reference below Figure 4, which shows a hardware structure diagram of a head-mounted display device suitable for implementing some embodiments of the present disclosure. Figure 4 The head mounted display device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0112] like Figure 4 As shown, the AR device 400 includes a processing device (such as a central processing unit, an image processor, etc.) 401, a memory (ROM) 402, an input unit 403 and an output unit 404, wherein the processing device 401, the memory 402, the input unit 403 and the output unit 404 are connected to each other through a bus 405. Here, the method according to some embodiments of the present disclosure can be implemented as a computer program and stored in the memory 402. The processing device 401 in the AR device 400 specifically implements the interface interaction function of the 3D space defined in the method of some embodiments of the present disclosure by calling the above-mentioned computer program stored in the memory 402. In some implementations, the input unit 403 may include devices such as a camera, a microphone, a gyroscope, an accelerometer, a magnetometer, etc., and the output unit 404 may be a display module or other device that can be used to display content. The above-mentioned display module may include an optical machine and an optical element. The above-mentioned optical element may include an optical element such as a prism, a free-form surface, a BirdBath, and an optical waveguide. Therefore, when the processing device 401 calls the above-mentioned computer program to execute the interface interaction function of the 3D space, it can control the input unit 403 to obtain the user's gesture, voice and other operation instructions, and control the output unit 404 to display the display content.

[0113] It should be noted that the computer-readable medium in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, the computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0114] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0115] The computer-readable medium may be included in the electronic device; or it may exist independently without being assembled into the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains the posture information of the head-mounted display device, the target screen frame to be rendered, and the shared texture data of the target application process; determines the distortion correction information of the target screen frame to be rendered according to the shared texture data and the posture information; in response to detecting that the historical augmented reality screen recording video is jittery, the posture information is smoothed in the time dimension to obtain the smoothed posture information, wherein the historical augmented reality screen recording video is the screen recording video corresponding to each screen frame to be rendered before the target screen frame to be rendered; according to the distortion correction information, the smoothed posture information, and the shared texture data, the target screen frame to be rendered is rendered as a whole to obtain a rendered screen frame, so that the server can perform screen recording processing on the historical augmented reality screen recording video and the rendered screen frame to obtain an augmented reality screen recording video.

[0116] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0117] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0118] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0119] Some embodiments of the present disclosure further provide a computer program product, including a computer program, which implements any of the above-mentioned rendering screen recording methods when executed by a processor.

[0120] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) and the technical solutions formed.

Claims

1. A method for recording a rendered image, comprising: Obtaining the posture information of the head mounted display device, the target frame to be rendered, and the shared texture data of the target application process; Determining distortion correction information of the target to-be-rendered picture frame according to the posture information; In response to detecting that jitter occurs in the historical augmented reality screen recording video, the position information is smoothed in the time dimension to obtain smoothed position information, wherein the historical augmented reality screen recording video is a screen recording video corresponding to each to-be-rendered screen frame that is located before the target to-be-rendered screen frame; According to the distortion correction information, the smoothed posture information and the shared texture data, the target screen frame to be rendered is rendered as a whole to obtain a rendered screen frame, so that the server can perform screen recording processing on the historical augmented reality screen recording video and the rendered screen frame to obtain an augmented reality screen recording video.

2. The method according to claim 1, wherein: The method further comprises: In response to detecting that the rendering of the target application process is completed and the current application process meets the conditions for starting the rendering operation, the shared texture data corresponding to the current application process, the posture information of the head-mounted display device and the target to-be-rendered screen frame are obtained to perform overall rendering of the screen.

3. The method according to claim 1, wherein: Determining the distortion correction information of the target to-be-rendered picture frame according to the posture information includes: Acquire device calibration information and screen display position information of the head mounted display device, wherein the screen display position information is position information extracted from the posture information at the previous time point; Determine a first position difference between the screen display position information and the posture information; Determining target mapping display position information of the target to-be-rendered picture frame according to the first position difference; Performing anti-distortion fusion processing on the device calibration information and the first position difference to obtain a second position difference; The target mapping display position information and the second position difference are determined as distortion correction information.

4. The method according to claim 1, wherein: The step of performing time dimension smoothing on the posture information to obtain smoothed posture information includes: Determine average pose information of the target picture frame to be rendered and the previous picture frame to be rendered; Determining a picture rotation angle according to the average posture information; Determine the product of the derivative of the picture rotation angle with respect to time and the angular velocity in the posture information as the picture offset; Determining the picture smoothness of the target picture frame to be rendered according to the picture offset; Smoothed posture information is generated according to the picture smoothness and the average posture information.

5. The method according to claim 1, wherein: The method further comprises: Controlling the low-latency rendering thread included in the target application process to perform low-latency parallel rendering on the target to-be-rendered picture frame according to the shared texture data to obtain a low-latency rendered picture frame; The obtained low-latency rendered picture frames are determined as screen projection videos, wherein the low-latency rendered picture frames are picture frames obtained through low-latency rendering.

6. The method according to claim 1, wherein: The step of obtaining the posture information of the head mounted display device, the target picture frame to be rendered, and the shared texture data of the target application process includes: Control the low-latency rendering thread and the screen recording rendering thread included in the target application process to share an open image library context, wherein the open image library context is the rendering state and resource information required for rendering created by the low-latency rendering thread or the screen recording rendering thread; Obtaining shared texture data through the open image library context; Acquiring posture information through the head-mounted display device; Obtain a target to-be-rendered screen frame corresponding to the target application process.

7. The method according to claim 1, wherein: The augmented reality screen recording video is obtained by the following steps: In response to determining to establish a communication connection with the target application process through an image canvas communication mechanism, obtaining an initial rendered screen frame corresponding to the target application process, and starting a recording operation, wherein the initial rendered screen frame is the first rendered screen frame in the historical augmented reality screen recording video; In response to detecting that the recording operation is successfully started, the historical augmented reality screen recording video and the rendered picture frame are video encoded to obtain an augmented reality screen recording video bit stream as the augmented reality screen recording video.

8. A head mounted display device, comprising: one or more processors; A storage device for storing one or more programs; at least one display screen and optical element for forming an image in front of the user's eyes; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

9. A computer readable medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images