Unity3D-based co-focusing rendering methods, systems, and electronic devices

Abstract

This application provides a Unity3D-based joint focus rendering method, system, and electronic device, relating to the field of rendering technology, and alleviates the technical problem of low image quality when rendered by existing frameworks. The method includes: a mobile device sending camera position, rotation parameters, and eye-tracking data to the cloud via a local area network; the cloud rendering an outer layer of the image based on the camera position, rotation parameters, and eye-tracking data, obtaining an outer layer rendering result, encoding the outer layer rendering result, obtaining an outer layer encoded result, and sending the outer layer encoded result to the mobile device via the local area network; the mobile device rendering a focus layer of the image based on the eye-tracking data, obtaining a focus layer rendering result, decoding the received outer layer encoded result, obtaining an outer layer decoded result, and stitching the outer layer decoded result and the focus layer rendering result together to generate and display the stitched image result.

Description

Technical Field

[0001] This application relates to the field of rendering technology, and in particular to a joint focus rendering method, system, and electronic device based on Unity3D. Background Technology

[0002] Currently, Unity3D (3D) is a common tool for developing Virtual Reality (VR) applications. Unity3D currently only supports three frameworks: "full-screen mobile local rendering," "full-screen cloud remote rendering," and "mobile local focus rendering." However, rendering using these existing frameworks results in lower image quality for the final product, leading to a less than ideal visual experience for VR users. Summary of the Invention

[0003] The purpose of this invention is to provide a Unity3D-based co-focus rendering method, system, and electronic device to alleviate the technical problem of low image quality when rendered by existing frameworks and presented to users.

[0004] In a first aspect, embodiments of this application provide a joint focus rendering method based on Unity3D, the method comprising:

[0005] The first script on the mobile device sends the camera position, rotation parameters, and eye-tracking data to the cloud via the local area network through the first function.

[0006] The first script in the cloud blocks and waits through the first function until it receives the camera position, the rotation parameters, and the eye tracker capture data. Then, both the cloud and the mobile device enter the first function of the second script.

[0007] The cloud platform renders the outer layer of the image based on the camera position, the rotation parameters, and the eye tracker capture data, obtains the outer layer rendering result, encodes the outer layer rendering result through a second function to obtain the outer layer encoding result, and sends the outer layer encoding result to the mobile terminal through the local area network.

[0008] The mobile device renders the image focus layer based on the eye tracker's captured data, obtains the focus layer rendering result, blocks and waits for the outer layer encoding result transmitted from the cloud, and decodes the received outer layer encoding result through the second function to obtain the outer layer decoding result. Based on the outer layer decoding result and the focus layer rendering result, the image stitching result is generated and displayed.

[0009] In one possible implementation, the step of the mobile device rendering an image focus layer based on the eye-tracking data to obtain the focus layer rendering result includes:

[0010] The mobile device renders the image focus layer by modifying the camera projection matrix based on the center position and radius parameters of the focus layer captured by the eye tracker, thus obtaining the focus layer rendering result.

[0011] In one possible implementation, the step of the mobile terminal rendering the image focus layer by modifying the camera projection matrix based on the center position and radius parameters of the focus layer captured by the eye tracker, and obtaining the focus layer rendering result, includes:

[0012] The mobile device normalizes some parameters corresponding to the full-screen rendering state, and calculates new parameters corresponding to the image focus layer only based on the normalization result and the center position and radius parameters of the focus layer in the eye tracker capture data. Based on the new parameters, the camera projection matrix is ​​modified to obtain the rendering result for the image focus layer.

[0013] In one possible implementation, the step of encoding the outer layer rendering result using a second function to obtain the outer layer encoding result includes:

[0014] The cloud uses an H.264 encoder to encode the outer layer rendering result through a second function, and keeps each frame encoded by the H.264 encoder as an I-frame to obtain the outer layer encoding result.

[0015] In one possible implementation, the step of sending the outer layer encoding result to the mobile terminal via a local area network includes:

[0016] During the process of encoding the outer layer rendering result using the H.264 encoder in the cloud through the second function, the GOP attribute is set to 0 so that the outer layer encoding result contains only I-frames, and a specified ultra-fast mode is set to reduce the encoding latency in the cloud and the decoding latency in the mobile end, and a specified zero-latency mode is set to allow the encoder to release the encoded data.

[0017] After obtaining the encoded data byte array corresponding to the outer layer encoding result, the Write method is used to send the encoded data byte array to the IP address of the mobile terminal via the local area network.

[0018] In one possible implementation, the step of decoding the received outer layer encoding result using the second function to obtain the outer layer decoding result includes:

[0019] After receiving the outer layer encoding result transmitted from the cloud, the mobile device directly uploads the outer layer encoding result to the GPU;

[0020] After the GPU decodes the outer layer encoding result, it directly accesses the outer layer decoding result in the GPU through the shader.

[0021] In one possible implementation, the step of stitching together the outer layer decoding result and the focus layer rendering result to generate and display the image stitching result includes:

[0022] The mobile device uses the textures of the outer layer and the focus layer of the image as input through the GPU shader, and calculates the focus layer region of the texture to be sampled and the outer layer region of the texture to be sampled on the screen based on the center position and radius parameters of the focus layer captured by the eye tracker. The image stitching result is generated and displayed based on the focus layer region and the outer layer region. The GPU shader uses the OpenGL shading language program.

[0023] Secondly, a Unity3D-based co-focusing rendering system is provided, including mobile and cloud-based versions;

[0024] The mobile device is used to send camera position, rotation parameters, and eye tracker captured data to the cloud via a local area network using a first script and a first function.

[0025] The cloud is used to block and wait through the first function using the first script until the camera position, the rotation parameters, and the eye tracker capture data are received. Both the cloud and the mobile terminal enter the first function of the second script.

[0026] The cloud is also used to render the outer layer of the image based on the camera position, the rotation parameters and the eye tracker capture data, to obtain the outer layer rendering result, and to encode the outer layer rendering result through the second function to obtain the outer layer encoding result, and to send the outer layer encoding result to the mobile terminal through the local area network;

[0027] The mobile terminal is also used to render the image focus layer based on the eye tracker captured data, obtain the focus layer rendering result, block and wait for the outer layer encoding result transmitted from the cloud, and decode the received outer layer encoding result through the second function to obtain the outer layer decoding result. Based on the outer layer decoding result and the focus layer rendering result, the mobile terminal is stitched together to generate and display the image stitching result.

[0028] Thirdly, embodiments of this application provide an electronic device, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the processor executes the computer program to implement the method described in the first aspect above.

[0029] Fourthly, embodiments of this application also provide a computer-readable storage medium storing computer-executable instructions, which, when invoked and executed by a processor, cause the processor to perform the method described in the first aspect above.

[0030] The embodiments of this application bring the following beneficial effects:

[0031] This application provides a Unity3D-based joint focus rendering method, system, and electronic device. A first script on a mobile device sends camera position, rotation parameters, and eye-tracking data to the cloud via a local area network (LAN) through a first function. A first script on the cloud blocks and waits through the first function until it receives the camera position, rotation parameters, and eye-tracking data. Both the cloud and the mobile device then enter the first function of a second script. The cloud renders the outer layer of an image based on the camera position, rotation parameters, and eye-tracking data to obtain an outer layer rendering result. It then encodes the outer layer rendering result using a second function to obtain an outer layer encoded result, which is sent to the mobile device via the LAN. The mobile device renders the focus layer of an image based on the eye-tracking data to obtain a focus layer rendering result. It blocks and waits for the outer layer encoded result transmitted from the cloud and decodes the received outer layer encoded result using the second function to obtain an outer layer decoded result. The outer layer decoded result and the focus layer rendering result are then stitched together to generate and display an image stitching result. In this solution, an eye tracker captures the user's eye position on the screen. The mobile device only renders the small area on the screen that the user is looking at, i.e., the focused layer, while the rendering of other areas on the screen, i.e., the outer layer, is done by the cloud. After the cloud rendering is completed, it is sent back to the mobile device via the local area network for stitching, and finally the content of the frame is presented to the user. This joint focus rendering method can help the mobile device provide users with high-quality virtual reality (VR) applications with limited graphics rendering capabilities, i.e., the quality of the final image presented to the user is improved.

[0032] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0034] Figure 1 A flowchart illustrating the Unity3D-based joint focus rendering method provided in this application embodiment;

[0035] Figure 2 A block diagram illustrating the implementation of the Unity 3D-based joint rendering framework provided in this application embodiment;

[0036] Figure 3 A schematic diagram of a camera's view frustum provided in an embodiment of this application;

[0037] Figure 4 This application provides pseudocode for hardware decoding on a Unity3D mobile device.

[0038] Figure 5 This application provides pseudocode for a GPU shader used during splicing and rendering, as exemplified in this application.

[0039] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0041] The terms "comprising" and "having," and any variations thereof, used in the embodiments of this application, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the steps or units listed, but may optionally include other steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0042] Currently, developers are unable to quickly apply joint focus rendering technology to VR application development. Rendering using the three existing frameworks results in low-quality images presented to users, leading to a poor VR user experience.

[0043] Based on this, this application provides a Unity3D-based joint focus rendering method, system, and electronic device, which can alleviate the technical problem of low image quality when rendered by existing frameworks and presented to users.

[0044] The embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0045] Figure 1 This is a flowchart illustrating a Unity3D-based joint focus rendering method provided in an embodiment of this application. Figure 1 As shown, the method includes:

[0046] In step S110, the first script on the mobile terminal sends the camera position, rotation parameters, and eye-tracking data to the cloud via the local area network through the first function.

[0047] In practical applications, a VR application can be created in Unity3D, ensuring that the server (i.e., the cloud) and client (i.e., the mobile device) are on the same local area network. The Unity3D project is then compiled, and the VR application is deployed to the mobile device. Simultaneously launching the VR application on both the cloud and mobile devices allows for the experience of joint focus rendering. The processing flow for each frame during VR application runtime can utilize the methods provided in the embodiments of this application.

[0048] In this step, such as Figure 2 As shown, the PlayerControl.cs script (i.e., the first script) on the mobile device sends the camera's position and rotation parameters, the center position and radius of the focusing layer captured by the eye tracker to the cloud via the local area network in the Update() function (i.e., the first function).

[0049] In step S120, the first script in the cloud blocks and waits through the first function until it receives the camera position, rotation parameters, and eye-tracking data. Then, both the cloud and the mobile device enter the first function of the second script.

[0050] The implementation block diagram of the Unity3D-based joint rendering framework is as follows: Figure 2 As shown, this framework is based on the UnityMonoBehavior class and includes two scripts: PlayerControl.cs (the first script) and CollaborativeFoveatedRendering.cs (the second script). Both scripts need to be attached to the Unity3D camera component. To achieve collaborative rendering, the same VR application needs to be deployed on both the cloud and mobile devices. The two scripts can determine whether to execute the cloud processing flow or the mobile processing flow based on the IP address.

[0051] As one possible implementation, modify the settings for mobile and cloud IP addresses in PlayerControl.cs (i.e., the first script) and CollaborativeFoveatedRendering.cs (i.e., the second script) to add a camera component to the user-controlled character in the VR application scenario, such as... Figure 2 As shown, it also mounts the two scripts PlayerControl.cs (i.e., the first script) and CollaborativeFoveatedRendering.cs (i.e., the second script).

[0052] In this step, PlayerControl.cs (the first script) in the cloud blocks and waits in the Update() function (the first function) until it receives the aforementioned parameters: camera position, rotation parameters, and eye-tracking data. Both the cloud and mobile devices then enter the Update() function (the first function) of CollaborativeFoveatedRendering.cs (the second script).

[0053] In step S130, the cloud renders the outer layer of the image based on the camera position, rotation parameters, and eye-tracking data to obtain the outer layer rendering result. The outer layer rendering result is then encoded using a second function to obtain the outer layer encoding result, which is then sent to the mobile device via the local area network.

[0054] In this step, such as Figure 2 As shown, the cloud begins rendering the outer layer. After rendering, it enters the OnRenderImage() function (i.e., the second function) to encode the result and send the encoded result to the mobile device via the local area network.

[0055] In step S140, the mobile terminal renders the image focus layer based on the data captured by the eye tracker, obtains the focus layer rendering result, blocks and waits for the outer layer encoding result to be transmitted from the cloud, and decodes the received outer layer encoding result through the second function to obtain the outer layer decoding result. Based on the outer layer decoding result and the focus layer rendering result, the image stitching result is generated and displayed.

[0056] As one possible implementation, while performing step S130 as described above, such as Figure 2 As shown, the mobile device can start rendering the focus layer. After rendering, it will block and wait for the outer layer encoding result to be sent back from the cloud. Then, it will enter the OnRenderImage() function (i.e. the second function) to decode the encoding result. Finally, it will complete the splicing and presentation with the focus layer rendering result.

[0057] In practical applications, such as Figure 2As shown, the mutual waiting relationship between the mobile device and the cloud ensures that the focus layer and the outer layer always belong to the same frame in the joint focus rendering, and no misalignment will occur.

[0058] In this embodiment of the application, for each frame during the operation of the VR application, an eye tracker captures the position of the user's eyes focused on the screen, such as... Figure 2 As shown, the mobile device renders only the small area on the screen that the user is looking at (i.e., the focus layer), while the rendering of other areas on the screen (i.e., the outer layer) is done in the cloud. After rendering is completed in the cloud, the data is sent back to the mobile device via the local area network for stitching, and finally the content of that frame is presented to the user. This joint focus rendering method can help mobile devices provide users with high-quality virtual reality (VR) applications with limited graphics rendering capabilities.

[0059] The Unity3D-based joint focus rendering method helps developers quickly deploy joint focus rendering technology to VR applications. This method provides developers with two extensible scripts to achieve joint focus rendering effects. These scripts dynamically adjust the camera projection matrix based on eye-tracking results, enabling mobile devices to correctly render the focus layer. Simultaneously, they perform real-time GPU hardware decoding of the encoded results of the cloud-based peripheral layer and correctly stitch them with the locally rendered focus layer.

[0060] Furthermore, through the Unity MonoBehavior-based joint focus rendering scripts, namely the first and second scripts mentioned above, developers can use joint focus rendering technology without worrying about implementation details, and can also extend custom functionality within the scripts.

[0061] The steps described above will be explained in detail below.

[0062] In some embodiments, the process of rendering the image focus layer based on eye-tracking data and obtaining the focus layer rendering result in step S140 above may include the following steps:

[0063] Step a) The mobile device renders the image focus layer by modifying the camera projection matrix based on the center position and radius parameters of the focus layer captured by the eye tracker, and obtains the focus layer rendering result.

[0064] In practical applications, this method can adjust the projection matrix of the Unity3D camera component for each frame of a game, based on the changing center position and radius of the focus layer, to ensure correct focus layer rendering on mobile devices. In other words, it dynamically adjusts the camera projection matrix based on eye-tracking results, enabling accurate focus layer rendering on mobile devices.

[0065] In this embodiment, by modifying the camera projection matrix based on the center position and radius parameters of the focusing layer returned by the eye tracker, the focusing rendering effect corresponding to the parameters can be correctly achieved.

[0066] Based on step a) above, step a) may include the following steps:

[0067] Step b) The mobile device normalizes some parameters corresponding to the full-screen rendering state, and calculates new parameters corresponding to the image focus layer based on the normalization result and the center position and radius parameters of the focus layer in the eye tracker capture data. Based on the new parameters, the camera projection matrix is ​​modified to obtain the rendering result for the image focus layer.

[0068] Regarding the rendering of the focus layer on mobile devices, it should be noted that the view frustum of the camera component in Unity3D is as follows: Figure 3 As shown, this includes the Near Clip Plane (NCP) and the Far Clip Plane (FCP). Only objects located between the NCP and the FCP are rendered. The absolute value of the distance from the NCP to the camera is n, and the absolute value of the distance from the FCP to the camera is f. The size of the mobile screen is normalized to four values: top t, bottom b, left l, and right r. The projection matrix of the camera component is determined by the six parameters n, f, t, b, l, and r mentioned above. In this embodiment, to change the mobile screen from rendering the full screen to rendering only the focus layer, the four normalized parameters t, b, l, and r corresponding to rendering the full screen are combined with the center position and radius of the focus layer captured by the eye tracker to recalculate the new four parameters t, b, l, and r corresponding to rendering only the focus layer. These parameters are then updated in the projection matrix, thereby efficiently achieving the effect of rendering only the focus layer.

[0069] In some embodiments, the process of encoding the outer layer rendering result using the second function to obtain the outer layer encoding result in step S130 above may include the following steps:

[0070] Step c) The cloud uses the H.264 encoder to encode the outer layer rendering result through the second function, and keeps each frame encoded by the H.264 encoder as an I-frame to obtain the outer layer encoding result.

[0071] Regarding the cloud-based encoding of the outer layer, it's important to note that in H.264, traditional B and P frame decoding requires the participation of historical and / or future frames, increasing cache pressure and decoding latency. However, in this embodiment, the cloud uses full I-frame encoding of the outer layer rendering result, allowing the mobile device to decode the encoded result immediately upon receipt, eliminating the need for caching. At a 600x540 outer layer resolution, 60FPS full I-frame encoding requires only 1.32MB / s of local area network bandwidth. In this embodiment, the encoding result does not require caching, reducing cache pressure and decoding latency.

[0072] Based on step c) above, the process of sending the outer layer encoding result to the mobile terminal via the local area network in step S130 above may include the following steps:

[0073] Step d), during the process of encoding the outer layer rendering result using the H.264 encoder in the cloud through the second function, the GOP attribute is set to 0 so that the outer layer encoding result contains only I-frames, and a specified ultra-fast mode is set to reduce the encoding latency in the cloud and the decoding latency on the mobile end, and a specified zero-latency mode is set to allow the encoder to release the encoded data.

[0074] Step e) After obtaining the encoded data byte array corresponding to the outer layer encoding result, the encoded data byte array is sent to the IP address of the mobile terminal via the local area network using the Write method.

[0075] It should be noted that the cloud-based encoding outer layer rendering results use FFmpeg integrated in C#. An H.264 encoder is employed, with three types of encoded frames: I-frames, P-frames, and B-frames. Only I-frames can be decoded independently; decoding P-frames requires historical frames, and decoding B-frames requires both historical and future frames.

[0076] In this embodiment, it is ensured that each frame encoded by H.264 is an I-frame, so that the mobile device can decode it immediately after receiving the encoding result from the cloud outer layer. The "Group of Pictures (GOP)" attribute determines the distance between adjacent I-frames, so in this embodiment, this attribute needs to be set to 0 to ensure that the encoding result contains only I-frames. Then, the "Ultra-Fast" mode needs to be set to reduce the latency between cloud encoding and mobile decoding. Finally, the "Zero Latency" mode needs to be set to allow the encoder to immediately release the encoded data, rather than saving it for several frames. After obtaining the encoded data byte array, it is sent to the client IP address using the "Write" method of the C# NetworkStream class.

[0077] In practical applications, the sum of the additional latency introduced by projection matrix calculation, I-frame encoding, hardware decoding, and stitching operations in the joint rendering framework, as well as the rendering latency, can be less than 16.66ms under certain constraints, meeting the requirement of up to 60FPS for mobile applications.

[0078] In some embodiments, the process of decoding the received outer layer encoding result using the second function in step S140 to obtain the outer layer decoding result may include the following steps:

[0079] Step f): After receiving the outer layer encoding result transmitted from the cloud, the mobile device directly uploads the outer layer encoding result to the GPU.

[0080] Step g) After the GPU decodes the outer layer encoding result, it directly accesses the outer layer decoding result in the GPU through the shader.

[0081] Regarding the decoding of the outer layer on mobile devices, it's important to note that traditional software decoding is performed on the CPU. The decoding result is then uploaded to the GPU shader for post-processing and finally drawn on the screen. However, the decoding result reverts to the data volume of the initial rendered frame, and uploading it to the GPU introduces significant latency. GPU hardware decoding may be comparable to CPU software decoding in speed, but its advantage lies in its efficient data flow. The mobile device directly uploads the received cloud-based outer layer encoding result to the GPU. The data volume of the encoding result is small, so the time consumption of this step can be ignored. In this embodiment, after the GPU performs decoding, the shader can directly access the decoding result on the GPU, avoiding the latency of uploading the software decoding result to the GPU and resulting in a highly efficient data flow.

[0082] Hardware decoding is performed using the Android hardware decoding library Mediacodec. This library uses the Java programming language, but Unity 3D uses C#. Therefore, in this embodiment, the "AndroidJavaClass" API provided by Unity3D is used to call the Java hardware decoding library from C#. Figure 4This demonstrates pseudocode for performing mobile hardware decoding in a C# script using Unity 3D. First, a Java class, `AsyncMediaCodecPlugIn`, is provided for hardware decoding. Its constructor initializes the MediaCodec hardware decoder asynchronously, creating an OpenGL texture on the GPU and setting it as the decoder's output. Then, the `AsyncMediaCodecPlugIn` constructor is called in the C# script using Unity 3D's "AndroidJavaClass" API, and the created OpenGL texture is linked via a pointer using Unity 3D's "CreateExternalTexture" API. This allows the GPU shader to directly access the decoding result through the OpenGL texture linked on the GPU. At a 600x540 outer layer resolution, hardware decoding takes an average of only 2ms.

[0083] In this embodiment, the mobile device uses the Android MediaCodec hardware decoding library to directly decode the outer layer cloud rendering results on the mobile device GPU. This avoids the additional latency caused by uploading the traditional software decoding results to the GPU, and achieves fast, real-time and efficient decoding of the outer layer encoding results in the cloud using GPU hardware.

[0084] In some embodiments, the process of stitching together the outer layer decoding result and the focus layer rendering result in step S140 above to generate and display the image stitching result may include the following steps:

[0085] Step h): The mobile device takes the textures of the outer layer and the focus layer of the image as input through the GPU shader, and calculates the focus layer part of the texture of the focus layer to be sampled and the outer layer part of the texture of the outer layer to be sampled on the screen according to the center position and radius parameters of the focus layer captured by the eye tracker. Based on the focus layer part and the outer layer part, the image stitching result is generated and displayed.

[0086] The GPU shader utilizes the OpenGL Shading Language program. It should be noted that, since the outer layer texture is an OpenGL texture, the GPU shader in this embodiment should use the OpenGL Shading Language (GLSL) program.

[0087] Regarding the process of stitching and presenting the rendering results of the focus layer and the outer layer, it should be noted that GPU shaders are often used for post-processing of the final image displayed on the screen. The stitching between the focus layer and the outer layer is actually a form of post-processing. In this embodiment, the GPU shader takes the focus layer and outer layer textures as input and calculates which part of the screen should sample the focus layer texture and which part should sample the outer layer texture based on the center position and radius parameters of the focus layer captured by the eye tracker. Since the outer layer texture is an OpenGL texture, the GPU shader should use an OpenGL Shading Language (GLSL) program.

[0088] In practical applications, a complete GLSL shader program contains vertex methods and fragment methods. In the post-processing context, the vertex method returns regularized screen coordinates, which the fragment method can use to sample the texture. Although each fragment method is only responsible for one pixel value on the screen, it will execute in parallel on the GPU so that all screen pixel values ​​can be computed simultaneously. Therefore, a common fragment method needs to be written for all screen pixel values, its pseudocode as follows: Figure 5 As shown. For each screen pixel, the regularized distance between the regularized pixel coordinates obtained by the vertex method and the center position of the focus layer is different. Therefore, it is necessary to calculate this distance `dist` using the `distance()` method and compare it with the radius of the focus layer to determine whether the texture to be sampled by the `resampleTexture()` method comes from the focus layer or the outer layer. Specifically, if `dist` is less than the radius of the focus layer, it means that this part of the screen needs to display the focus layer result, and the `resampleTexture()` method is used to sample the focus layer texture. Otherwise, it means that this part of the screen needs to display the outer layer result, and the `resampleTexture()` method is used to sample the outer layer texture.

[0089] In this embodiment, a function is provided to correctly stitch the rendering results of the mobile terminal's focus layer and the outer layer at any center position and radius, thereby enabling the focus layer and the outer layer to be correctly stitched together and avoiding any misalignment.

[0090] This application embodiment also provides a Unity3D-based co-focus rendering device, which includes: a mobile terminal and a cloud terminal;

[0091] The mobile device is used to send camera position, rotation parameters, and eye tracker captured data to the cloud via a local area network using a first script and a first function.

[0092] The cloud is used to block and wait through the first function using the first script until the camera position, the rotation parameters, and the eye tracker capture data are received. Both the cloud and the mobile terminal enter the first function of the second script.

[0093] The cloud is also used to render the outer layer of the image based on the camera position, the rotation parameters and the eye tracker capture data, to obtain the outer layer rendering result, and to encode the outer layer rendering result through the second function to obtain the outer layer encoding result, and to send the outer layer encoding result to the mobile terminal through the local area network;

[0094] The mobile terminal is also used to render the image focus layer based on the eye tracker captured data, obtain the focus layer rendering result, block and wait for the outer layer encoding result transmitted from the cloud, and decode the received outer layer encoding result through the second function to obtain the outer layer decoding result. Based on the outer layer decoding result and the focus layer rendering result, the mobile terminal is stitched together to generate and display the image stitching result.

[0095] In some embodiments, the mobile device is specifically used for:

[0096] Based on the center position and radius parameters of the focal layer in the eye-tracking data, the image focal layer is rendered by modifying the camera projection matrix to obtain the focal layer rendering result.

[0097] In some embodiments, the mobile device is specifically used for:

[0098] The parameters corresponding to the full-screen rendering state are normalized. Based on the normalization result and the center position and radius parameters of the focusing layer in the eye tracker capture data, new parameters corresponding to the focusing layer of the image are calculated. The camera projection matrix is ​​then modified based on the new parameters to obtain the rendering result for the focusing layer of the image.

[0099] In some embodiments, the cloud is specifically used for:

[0100] The outer layer rendering result is encoded using an H.264 encoder through a second function, while ensuring that each frame encoded by the H.264 encoder is an I-frame, thus obtaining the outer layer encoding result.

[0101] In some embodiments, the cloud is specifically used for:

[0102] During the process of encoding the outer layer rendering result using the H.264 encoder through the second function, the GOP attribute is set to 0 so that the outer layer encoding result contains only I-frames, and a specified ultra-fast mode is set to reduce the encoding latency of the cloud and the decoding latency of the mobile end, and a specified zero-latency mode is set to allow the encoder to release the encoded data.

[0103] After obtaining the encoded data byte array corresponding to the outer layer encoding result, the Write method is used to send the encoded data byte array to the IP address of the mobile terminal via the local area network.

[0104] In some embodiments, the mobile device is specifically used for:

[0105] After receiving the outer layer encoding result transmitted from the cloud, the outer layer encoding result is directly uploaded to the GPU; the GPU decodes the outer layer encoding result and then directly accesses the outer layer decoding result in the GPU through the shader.

[0106] In some embodiments, the mobile device is specifically used for:

[0107] The GPU shader takes the textures of the outer layer and the focused layer of the image as input, and calculates the focused layer region of the texture to be sampled and the outer layer region of the texture to be sampled on the screen based on the center position and radius parameters of the focused layer captured by the eye tracker. The image stitching result is generated and displayed by stitching together the focused layer region and the outer layer region. The GPU shader uses the OpenGL shading language program.

[0108] The Unity3D-based joint focus rendering device provided in this application embodiment has the same technical features as the Unity3D-based joint focus rendering method provided in the above embodiments, so it can also solve the same technical problems and achieve the same technical effects.

[0109] An electronic device provided in this application embodiment, such as Figure 6 As shown, the electronic device 600 includes a processor 602 and a memory 601. The memory stores a computer program that can run on the processor. When the processor executes the computer program, it implements the steps of the method provided in the above embodiments.

[0110] See Figure 6 The electronic device also includes a bus 603 and a communication interface 604. The processor 602, the communication interface 604 and the memory 601 are connected through the bus 603. The processor 602 is used to execute executable modules, such as computer programs, stored in the memory 601.

[0111] The memory 601 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 604 (which can be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc.

[0112] Bus 603 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 6 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0113] The memory 601 is used to store programs. After receiving an execution instruction, the processor 602 executes the program. The method executed by the apparatus defined by the process disclosed in any of the preceding embodiments of this application can be applied to the processor 602 or implemented by the processor 602.

[0114] The processor 602 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 602 or by instructions in software form. The processor 602 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 601, and processor 602 reads the information from memory 601 and, in conjunction with its hardware, completes the steps of the above method.

[0115] Corresponding to the above-described Unity3D-based joint focus rendering method, this application embodiment also provides a computer-readable storage medium storing computer-executable instructions. When the computer-executable instructions are invoked and executed by a processor, the computer-executable instructions cause the processor to perform the steps of the above-described Unity3D-based joint focus rendering method.

[0116] The Unity3D-based joint focus rendering device provided in this application embodiment can be specific hardware on a device or software or firmware installed on the device. The implementation principle and technical effects of the device provided in this application embodiment are the same as those in the foregoing method embodiments. For the sake of brevity, any parts not mentioned in the device embodiment can be referred to the corresponding content in the foregoing method embodiments. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can all be referred to the corresponding processes in the above method embodiments, and will not be repeated here.

[0117] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0118] For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0119] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0120] In addition, the functional units in the embodiments provided in this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0121] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the Unity3D-based joint focus rendering method described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0122] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0123] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The protection scope of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application. All should be covered within the protection scope of this application. Therefore, the protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A joint focus rendering method based on Unity3D, characterized in that, The same application is deployed on both the mobile device and the cloud, and the application includes the same first script and the same second script; the first script contains a first function, and the second script contains the first function and a second function; the first script and the second script are processing flows that determine the execution of different branches on the mobile device or the cloud based on IP addresses; the method includes: The first script on the mobile device sends the camera position, rotation parameters, and eye-tracking data to the cloud via the local area network through the first function. The first script in the cloud blocks and waits through the first function until it receives the camera position, the rotation parameters, and the eye tracker capture data. Then, the cloud and the mobile terminal respectively enter the first function of their own local second script. The cloud-based system renders the outer layer of the image based on the camera position, the rotation parameters, and the eye-tracking data, obtains the outer layer rendering result, and encodes the outer layer rendering result through the second function of the second script to obtain the outer layer encoding result. The outer layer encoding result is then sent to the mobile terminal via the local area network. The mobile device renders the image focus layer based on the eye tracker's captured data, obtains the focus layer rendering result, blocks and waits for the outer layer encoding result transmitted from the cloud, and decodes the received outer layer encoding result through the second function to obtain the outer layer decoding result. Based on the outer layer decoding result and the focus layer rendering result, the image stitching result is generated and displayed.

2. The method according to claim 1, characterized in that, The step of the mobile terminal rendering the image focus layer based on the eye tracker-captured data to obtain the focus layer rendering result includes: The mobile device renders the image focus layer by modifying the camera projection matrix based on the center position and radius parameters of the focus layer captured by the eye tracker, thus obtaining the focus layer rendering result.

3. The method according to claim 2, characterized in that, The steps of the mobile terminal rendering the image focus layer by modifying the camera projection matrix based on the center position and radius parameters of the focus layer captured by the eye tracker to obtain the focus layer rendering result include: The mobile device normalizes some parameters corresponding to the full-screen rendering state, and calculates new parameters corresponding to the image focus layer only based on the normalization result and the center position and radius parameters of the focus layer in the eye tracker capture data. Based on the new parameters, the camera projection matrix is ​​modified to obtain the rendering result for the image focus layer.

4. The method according to claim 1, characterized in that, The step of encoding the outer layer rendering result through the second function of the second script to obtain the outer layer encoding result includes: The cloud uses an H.264 encoder to encode the outer layer rendering result through the second function of the second script, and keeps each frame encoded by the H.264 encoder as an I-frame to obtain the outer layer encoding result.

5. The method according to claim 4, characterized in that, The step of sending the outer layer encoding result to the mobile terminal via a local area network includes: During the process of encoding the outer layer rendering result using the H.264 encoder in the cloud through the second function, the GOP attribute is set to 0 so that the outer layer encoding result contains only I-frames, and a specified ultra-fast mode is set to reduce the encoding latency in the cloud and the decoding latency in the mobile end, and a specified zero-latency mode is set to allow the encoder to release the encoded data. After obtaining the encoded data byte array corresponding to the outer layer encoding result, the Write method is used to send the encoded data byte array to the IP address of the mobile terminal via the local area network.

6. The method according to claim 1, characterized in that, The step of decoding the received outer layer encoding result using the second function to obtain the outer layer decoding result includes: After receiving the outer layer encoding result transmitted from the cloud, the mobile device directly uploads the outer layer encoding result to the GPU; After the GPU decodes the outer layer encoding result, it directly accesses the outer layer decoding result in the GPU through the shader.

7. The method according to claim 1, characterized in that, The step of stitching together the outer layer decoding result and the focus layer rendering result to generate and display the image stitching result includes: The mobile device uses the textures of the outer layer and the focus layer of the image as input through the GPU shader, and calculates the focus layer region of the texture to be sampled and the outer layer region of the texture to be sampled on the screen based on the center position and radius parameters of the focus layer captured by the eye tracker. The image stitching result is generated and displayed based on the focus layer region and the outer layer region. The GPU shader uses the OpenGL shading language program.

8. A Unity3D-based joint focus rendering system, characterized in that, The same application is deployed on both the mobile terminal and the cloud. The application includes the same first script and the same second script. The first script contains a first function, and the second script contains the first function and a second function. The first script and the second script are processing flows that determine the execution of different branches on the mobile terminal or the cloud based on the IP address. The system includes: a mobile terminal and a cloud terminal; The mobile device is used to send camera position, rotation parameters, and eye tracker captured data to the cloud via a local area network using a first script and a first function. The cloud is used to block and wait through the first function using the first script until the camera position, the rotation parameters, and the eye tracker capture data are received. The cloud and the mobile terminal then enter the first function of their respective local second scripts. The cloud is also used to render the outer layer of the image based on the camera position, the rotation parameters and the eye tracker capture data, to obtain the outer layer rendering result, and to encode the outer layer rendering result through the second function of the second script to obtain the outer layer encoding result, and to send the outer layer encoding result to the mobile terminal through the local area network; The mobile terminal is also used to render the image focus layer based on the eye tracker captured data, obtain the focus layer rendering result, block and wait for the outer layer encoding result transmitted from the cloud, and decode the received outer layer encoding result through the second function to obtain the outer layer decoding result. Based on the outer layer decoding result and the focus layer rendering result, the mobile terminal is stitched together to generate and display the image stitching result.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions that, when invoked and executed by a processor, cause the processor to perform the method according to any one of claims 1 to 7.

Images