Method and system for quickly encoding image in 3D engine
By rendering the content of multiple cameras onto the same RenderTexture in the 3D engine and utilizing GPU encoding, the problem of slow transfer speed between video memory and system memory is solved, achieving efficient image encoding and multi-camera image processing.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HUIZHIAN INFORMATION TECH CO LTD
- Filing Date
- 2025-11-25
- Publication Date
- 2026-05-01
AI Technical Summary
In 3D engines, existing technologies suffer from slow data transfer speeds between video memory and system memory during image encoding, resulting in low CPU encoding performance and difficulty in meeting the demand for multiple cameras to acquire images at high frame rates simultaneously.
By rendering the content of multiple cameras onto the same RenderTexture, and using Unity's RenderTexture.GetNativeTexturePtr method to obtain the pointer to the underlying graphics library, converting it to CUDA memory, and encoding it into a preset format using GPU, video memory is reused to reduce resource consumption.
It improves image encoding efficiency, reduces video memory consumption, enhances the frame rate and stability of multi-camera image processing, and simplifies subsequent processing procedures.
Smart Images
Figure CN121962297A_ABST
Abstract
Description
Technical Field
[0001] This invention proposes a method and system for rapidly encoding images in a 3D engine, belonging to the field of image processing technology. Background Technology
[0002] In 3D engines, certain scenes require saving and exporting images for user applications such as algorithmic reasoning. Figure 1 and Figure 2 As shown, the processing flow in the prior art is as follows:
[0003] 1. Set the Camera's render target to the RenderTexture object.
[0004] 2. Read the rendered pixel data from the RenderTexture object into the Texture2D object.
[0005] 3. Encode the content of Texture2D into a JPEG image.
[0006] 4. (Optional) Save to disk
[0007] The above process has the following significant performance bottlenecks:
[0008] 1. RenderTexture object data is stored in the graphics card's video memory, while Texture2D object data is stored in system memory. The data transfer speed between the two is relatively slow (approximately 32GBps on PCIe 4.0), proving that the code can be constructed as follows... Figure 3 As shown:
[0009] 2. JPEG encoding on the CPU is extremely slow and difficult to perform in parallel encoding of images from multiple cameras simultaneously. Specifically, its performance metrics are as follows: Figure 4 As shown.
[0010] Due to the aforementioned issues, it is difficult to meet the requirement of 20 cameras simultaneously acquiring 720P images at 24FPS. Summary of the Invention
[0011] This invention provides a method and system for rapidly encoding images in a 3D engine to solve the aforementioned technical problems in the prior art. The technical solution adopted is as follows:
[0012] A method for rapidly encoding images in a 3D engine, the method comprising:
[0013] Render the content from multiple cameras onto the same RenderTexture, and differentiate the positions of the different cameras:
[0014] The underlying graphics library pointer of the RenderTexture object can be obtained through the RenderTexture.GetNativeTexturePtr method provided by Unity;
[0015] The graphics pointer is obtained through RenderTexture, the resource format is converted and mapped to CUDA memory, and the GPU is used to encode it into a preset format, reusing video memory to reduce resource consumption.
[0016] Furthermore, the content from multiple cameras is rendered onto the same RenderTexture, and the positions of different cameras are differentiated, including:
[0017] Render the content of multiple cameras onto the same RenderTexture, and define an integer variable i, initialized to 0, for counting and indexing;
[0018] Use a foreach loop to iterate through the array or collection of cameras named cameras and set the rendering for each camera;
[0019] Create a rectangular region and increment the value of the counter i so that the next camera can be processed in the next loop.
[0020] Furthermore, rendering settings are configured for each camera, including:
[0021] For each camera, check if its label is "UnitCamera"; if the label does not match, skip the current loop iteration and continue to the next camera.
[0022] Set the targetTexture property of the current camera to camRenderTex, where camRenderTex is a rendering target texture;
[0023] Set the current camera's rendering path to RenderingPath.Forward, instructing the camera to use forward rendering;
[0024] Set the camera's clear flag to CameraClearFlags.SolidColor, and fill the background with a single color for each frame rendered.
[0025] Set the camera's background color to white;
[0026] Call the camera's SetReplacementShader method to set the effect shader;
[0027] Find the child object "UnityDrone" of the current camera object and set it to inactive;
[0028] Calculate the modulus of the current index i with respect to the total number of columns totalCol, and assign the result to the variable col. Also, calculate the result of dividing the current index i by the total number of columns totalCol, and assign it to the variable row.
[0029] Further, create a rectangular region, including:
[0030] Create a new rectangle object cam.rect using the Rect struct, defining its position and size:
[0031] The X-coordinate of the top-left corner is col * 1.0 / totalCol, representing the proportion it occupies in the column;
[0032] The Y-coordinate of the top-left corner is row * 1.0 / totalRow, representing the proportion it occupies in the row.
[0033] The width and height are both 1.0 / totalCol and 1.0 / totalRow, representing the proportion of each camera in the total columns and total rows.
[0034] Furthermore, by obtaining the graphics pointer through RenderTexture, converting the resource format and mapping it to CUDA memory, encoding it into a preset format using the GPU, and reusing video memory to reduce resource consumption, including:
[0035] By obtaining the pointer to the underlying graphics library, the graphics resources in Unity are converted into a resource format that meets the preset requirements;
[0036] The converted resource format is mapped to a two-dimensional memory resource cudaArray in CUDA using the CUAD method;
[0037] The two-dimensional memory resource cudaArray is converted into a uint8_t* type video memory pointer and input to the GPU-based JPEG encoding algorithm. The final output is a pointer to JPEG byte data that encodes the original pixel data into JPEG. The width and height of the JPEG image are verified. When the width and height of the current JPEG image input are the same as the previous JPEG image input, the original video memory resources are reused to reduce consumption.
[0038] A system for rapidly encoding images in a 3D engine, the system comprising:
[0039] The rendering position differentiation module is used to render the content of multiple cameras onto the same RenderTexture and differentiate the positions of different cameras:
[0040] The graphics library pointer acquisition module is used to obtain the underlying graphics library pointer of the RenderTexture object through the RenderTexture.GetNativeTexturePtr method provided by Unity.
[0041] The resource format conversion control module is used to obtain the graphics pointer through RenderTexture, convert the resource format and map it to CUDA memory, use GPU encoding to the preset format, and reuse video memory to reduce resource consumption.
[0042] Furthermore, the rendering position differentiation module includes:
[0043] The rendering control module is used to render the content of multiple cameras onto the same RenderTexture, and defines an integer variable i, initialized to 0, for counting and indexing;
[0044] The rendering settings module is used to iterate through an array or collection of cameras named cameras using a foreach loop and set the rendering settings for each camera.
[0045] The region creation module is used to create rectangular regions and increment the value of the counter i so that the next camera can be processed in the next loop.
[0046] Furthermore, the rendering settings module includes:
[0047] The label checking module checks whether the label of each camera is "UnitCamera"; if the label does not match, the current loop iteration is skipped and the process continues to the next camera.
[0048] The attribute setting module is used to set the targetTexture attribute of the current camera to camRenderTex, where camRenderTex is a rendering target texture;
[0049] The path setting module is used to set the current camera's rendering path to RenderingPath.Forward, instructing the camera to use forward rendering;
[0050] The background fill module is used to set the camera's clear flag to CameraClearFlags.SolidColor, and fill the background with a single color for each frame rendered.
[0051] The background color setting module is used to set the camera's background color to white;
[0052] The shader setting module is used to call the camera's SetReplacementShader method to set the effect shader;
[0053] The child object lookup module is used to find the child object "UnityDrone" of the current camera object and set it to inactive;
[0054] The row and column position determination module is used to calculate the modulus of the current index i with respect to the total number of columns totalCol, assign the result to the variable col, and calculate the result of dividing the current index i by the total number of columns totalCol, assigning it to the variable row;
[0055] Furthermore, the region creation module includes:
[0056] The rectangle object creation module is used to create a new rectangle object cam.rect using the Rect struct, defining its position and size:
[0057] The X-coordinate setting module is used to set the X-coordinate of the top left corner as col * 1.0 / totalCol, which represents the proportion occupied in the column;
[0058] The Y-coordinate setting module is used to set the Y-coordinate of the top left corner as row * 1.0 / totalRow, representing the proportion it occupies in the row.
[0059] The width and height setting module is used to set the width and height to 1.0 / totalCol and 1.0 / totalRow, respectively, representing the proportion of each camera in the total columns and total rows.
[0060] Furthermore, the resource format conversion control module includes:
[0061] The resource format conversion module is used to convert graphics resources in Unity into a resource format that meets preset requirements by obtaining the pointer to the underlying graphics library;
[0062] The mapping control module is used to map the converted resource format to a two-dimensional memory resource cudaArray in CUDA using the CUAD method;
[0063] The image output module is used to convert the two-dimensional memory resource cudaArray into a uint8_t* type video memory pointer and input it to the GPU-based JPEG encoding algorithm. The final output is a JPEG byte data pointer that encodes the original pixel data into JPEG. The module also verifies the width and height of the JPEG image. When the width and height of the current JPEG image input are the same as the previous JPEG image input, the original video memory resources are reused to reduce consumption.
[0064] Beneficial effects of this invention:
[0065] This invention proposes a method and system for rapidly encoding images in a 3D engine. It obtains the underlying graphics library pointer of the RenderTexture object using the Unity-provided RenderTexture.GetNativeTexturePtr method. For this invention, assuming Unity uses DX11 as its graphics library, it obtains an ID3D11Texture2D object. By obtaining the original graphics pointer, this invention utilizes methods provided by DX11 to convert graphics resources in Unity into resources that can be used subsequently. During this process, the original resource format (DXGI_FORMAT_R8G8B8A8_TYPELESS) is converted to a subsequently usable format (DXGI_FORMAT_R8G8B8A8_UNORM). The obtained ID3D11Texture2D resource is then mapped to a two-dimensional memory resource cudaArray in CUDA using CUDA methods (cudaGraphicsD3D11RegisterResource, cudaGraphicsMapResources). Using the cudaMemcpy2DFromArray method, this invention converts cudaArray resources into uint8_t type video memory pointers and inputs them into a GPU-based JPEG encoding algorithm. The final output is a pointer to JPEG byte data encoded from the original pixel data. During the process, various resources are validated based on the JPEG image width and height. If the previous width and height input are the same as the current input, the original video memory resources are reused to reduce consumption. The above code is written in C++, CUDA, and DX11 environments, compiled into a DLL, and used in Unity's C# implementation. The C# Marshal library is used to convert C pointers into C# byte arrays. Attached Figure Description
[0066] Figure 1 This is a schematic diagram of the present invention corresponding to the prior art;
[0067] Figure 2 The above is a code diagram corresponding to the prior art of this invention;
[0068] Figure 3 This invention provides code examples demonstrating the slower data transfer speed of Texture2D objects compared to existing technologies.
[0069] Figure 4 This is a schematic diagram of an image table of multiple cameras corresponding to the prior art of the present invention;
[0070] Figure 5 This is a schematic diagram corresponding to the method described in this invention;
[0071] Figure 6 The code corresponding to the method described in this invention Figure 1 ;
[0072] Figure 7 The code corresponding to the method described in this invention Figure 2 ;
[0073] Figure 8 The effect of the method described in this invention on the enemy camp Figure 1 ;
[0074] Figure 9 The effect of the method described in this invention on the enemy camp Figure 2 ;
[0075] Figure 10 The effect of the method described in this invention on the enemy camp Figure 3 . Detailed Implementation
[0076] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.
[0077] This invention proposes a method for rapidly encoding images in a 3D engine, such as... Figures 5 to 7 As shown, the method for quickly encoding images in a 3D engine includes:
[0078] Unlike the approach of rendering each camera onto a separate RenderTexture, this invention renders the content of multiple cameras onto the same RenderTexture and distinguishes the positions of different cameras:
[0079] The underlying graphics library pointer of the RenderTexture object can be obtained through the RenderTexture.GetNativeTexturePtr method provided by Unity;
[0080] The graphics pointer is obtained through RenderTexture, the resource format is converted and mapped to CUDA memory, and the GPU is used to encode it into a preset format, reusing video memory to reduce resource consumption.
[0081] The working principle and technical effect of the above technical solution are as follows: The underlying graphics library pointer of the RenderTexture object is obtained through the RenderTexture.GetNativeTexturePtr method provided by Unity. For this invention, assuming that Unity uses DX11 as its graphics library, the ID3D11Texture2D object is obtained. By obtaining the original graphics pointer, this invention uses the methods provided by DX11 to convert the graphics resources in Unity into resources that can be used subsequently. During this process, the original resource format (DXGI_FORMAT_R8G8B8A8_TYPELESS) is converted into a subsequently usable format (DXGI_FORMAT_R8G8B8A8_UNORM). The obtained ID3D11Texture2D resource is then mapped to a two-dimensional memory resource cudaArray in CUDA using CUDA methods (cudaGraphicsD3D11RegisterResource, cudaGraphicsMapResources). Using the cudaMemcpy2DFromArray method, this invention converts cudaArray resources into uint8_t type video memory pointers and inputs them into a GPU-based JPEG encoding algorithm. The final output is a pointer to JPEG byte data encoded from the original pixel data. During the process, various resources are validated based on the JPEG image width and height. If the previous width and height input are the same as the current input, the original video memory resources are reused to reduce consumption. The above code is written in C++, CUDA, and DX11 environments, compiled into a DLL, and used in Unity's C# implementation. The C# Marshal library is used to convert C pointers into C# byte arrays.
[0082] One embodiment of the present invention renders the content of multiple cameras onto the same RenderTexture and distinguishes the positions of different cameras, including:
[0083] Render the content of multiple cameras onto the same RenderTexture, and define an integer variable i, initialized to 0, for counting and indexing;
[0084] Use a foreach loop to iterate through the array or collection of cameras named cameras and set the rendering for each camera;
[0085] Create a rectangular region and increment the value of the counter i so that the next camera can be processed in the next loop.
[0086] The working principle of the above technical solution is as follows: This embodiment uses a single RenderTexture as a unified rendering target, and integrates and distinguishes the content of multiple cameras through the following logic: Define an integer variable i and initialize it to 0 to mark the index of the currently processed camera and calculate its corresponding rendering area; when traversing the cameras array, configure the rendering target of each camera as the same RenderTexture; dynamically calculate the rectangular area (such as position and size) of the camera on the RenderTexture based on the value of i, to ensure that the rendering content of each camera is limited to an independent area; after each loop, i is incremented to allocate a new area for the next camera and avoid content overlap.
[0087] The effects of the above technical solution are as follows: Figures 8 to 10 As shown, the camera content that originally required multiple rendering targets is integrated into a single RenderTexture, reducing resource consumption (such as memory and video memory consumption); the rectangular area division controlled by 'i' ensures that the content of each camera maintains spatial independence on the RenderTexture, forming a structured combined display; the unified rendering target reduces the overhead of rendering state switching, and subsequent processing of multi-camera content (such as post-processing and display) can be completed based on a single RenderTexture, simplifying the process; the rendering logic of each camera is not affected by other cameras, and its original content is completely preserved within the specified area, ensuring that no information is lost.
[0088] One embodiment of the present invention includes rendering settings for each camera, including:
[0089] For each camera, check if its label is "UnitCamera"; if the label does not match, skip the current loop iteration and continue to the next camera.
[0090] Set the targetTexture property of the current camera to camRenderTex, where camRenderTex is a rendering target texture;
[0091] Set the current camera's rendering path to RenderingPath.Forward, instructing the camera to use forward rendering;
[0092] Set the camera's clear flag to CameraClearFlags.SolidColor, and fill the background with a single color for each frame rendered.
[0093] Set the camera's background color to white (Color.white);
[0094] Call the camera's SetReplacementShader method to set the effects shader.
[0095] Find the child object "UnityDrone" of the current camera object and set it to inactive (i.e. disable the game object);
[0096] Calculate the modulus (remainder) of the current index i with respect to the total number of columns totalCol, and assign the result to the variable col. Also, calculate the result of dividing the current index i by the total number of columns totalCol, and assign it to the variable row. This will be used to determine the row and column positions of the camera in the grid.
[0097] Creating a rectangular region includes:
[0098] Create a new rectangle object cam.rect using the Rect struct, defining its position and size:
[0099] The X-coordinate of the top-left corner is col * 1.0 / totalCol, representing the proportion it occupies in the column;
[0100] The Y-coordinate of the top-left corner is row * 1.0 / totalRow, representing the proportion it occupies in the row.
[0101] The width and height are both 1.0 / totalCol and 1.0 / totalRow, representing the proportion of each camera in the total columns and total rows.
[0102] The working principle of the above technical solution is as follows: First, cameras are filtered by the label "UnitCamera", and only cameras that meet the conditions are processed, while irrelevant objects are filtered out; the filtered cameras are uniformly configured with rendering targets (camRenderTex), rendering paths (forward rendering), cleanup rules (solid color background fill), and background color (white) to ensure that the basic rendering properties are consistent; a unified effect shader is applied through SetReplacementShader to make the content of each camera present a unified style; the child object "UnityDrone" is disabled to exclude specific elements that do not need to be rendered, simplifying the screen; the row and column positions of each camera in the grid are determined based on the calculation of the current index i and the total number of columns totalCol and the total number of rows totalRow (col = i % totalCol, row = i / totalCol); a rectangular area is defined with proportional coordinates (col / totalCol, row / totalRow are the top left corners, 1 / totalCol, 1 / totalRow are the width and height), which limits the rendering content of each camera to the corresponding grid cell.
[0103] The effects of the above technical solution are as follows: Tag filtering processes only the target camera, reducing unnecessary calculations and improving rendering efficiency; unified rendering targets, paths, cleanup rules, and background colors ensure consistency in the underlying rendering logic of each camera's content, avoiding visual conflicts; the application of shader replacement ensures that all camera content presents the same special effects style, enhancing the overall picture coherence; disabling specific sub-objects and eliminating redundant elements allows the rendered content to focus more on core information; proportional rectangular areas based on row and column calculations enable the content of multiple cameras to form a regular grid arrangement on the RenderTexture, avoiding content overlap and ensuring layout adaptation (unaffected by the actual resolution of the RenderTexture), achieving orderly and compact multi-screen integration; unified target textures reduce rendering state switching overhead, providing convenience for subsequent secondary processing of the integrated picture (such as display and post-processing).
[0104] One embodiment of the present invention obtains a graphics pointer through RenderTexture, converts the resource format and maps it to CUDA memory, encodes it into a preset format using GPU, and reuses video memory to reduce resource consumption, including:
[0105] By obtaining the pointer to the underlying graphics library, the graphics resources in Unity can be converted into a resource format that meets the preset requirements using the methods provided by DX11; specifically: the original resource format (DXGI_FORMAT_R8G8B8A8_TYPELESS) is converted into a format that can be used later (DXGI_FORMAT_R8G8B8A8_UNORM).
[0106] The converted resource format is mapped to a two-dimensional memory resource cudaArray in CUDA using CUAD methods (cudaGraphicsD3D11RegisterResource, cudaGraphicsMapResources);
[0107] The two-dimensional memory resource cudaArray is converted into a uint8_t* type video memory pointer and input to the GPU-based JPEG encoding algorithm. The final output is a pointer to JPEG byte data that encodes the original pixel data into JPEG. The width and height of the JPEG image are verified. When the width and height of the current JPEG image input are the same as the previous JPEG image input, the original video memory resources are reused to reduce consumption.
[0108] The working principle of the above technical solution is as follows: Obtain the underlying graphics library pointer from RenderTexture; convert the original typeless format (DXGI_FORMAT_R8G8B8A8_TYPELESS) into a standardized unsigned normalized format (DXGI_FORMAT_R8G8B8A8_UNORM) based on the DX11 interface to ensure that the resource format meets the compatibility requirements of subsequent processing; utilize CUDA's graphics resource interaction methods (cudaGraphicsD3D11RegisterResource, cudaGraphicsMapResources) to map the converted graphics resources into CUDA-accessible two-dimensional memory resources (cudaArray), establishing a direct association between CPU-side graphics resources and GPU-side computing resources; convert the cudaArray into a uint8_t * type memory pointer, and pass it as input to the GPU-implemented JPEG encoding algorithm to complete the encoding conversion from pixel data to JPEG byte data; by verifying the consistency of the current JPEG image's width and height with the previous one, reuse the allocated memory resources when the conditions are met, avoiding repeated memory allocation and release operations.
[0109] The effects of the above technical solutions are as follows: format conversion ensures that graphics resources meet the format requirements of subsequent processing (such as CUDA access and encoding algorithms), eliminating processing anomalies caused by format incompatibility; CUDA mapping enables direct access of graphics resources to GPU memory, reducing data copy overhead between the CPU and GPU, and combined with GPU encoding algorithms, significantly improving encoding efficiency and breaking through the performance bottleneck of CPU encoding; the memory reuse mechanism avoids repeated memory allocation in scenes with the same width and height, reduces memory fragmentation and memory bandwidth occupation, and reduces latency caused by resource allocation and release; the entire process from graphics resource acquisition to JPEG encoding relies on GPU acceleration and resource reuse, shortening the overall processing cycle and enhancing the real-time performance and stability of the system in high frame rate and high resolution scenes.
[0110] In one embodiment of the present invention, the method for rapidly encoding images in a 3D engine further includes:
[0111] Real-time monitoring of the current camera's rendering operation parameters, including the number of frames rendered per unit time, rendering latency, and memory utilization.
[0112] The rendering operation coefficients corresponding to each unit of time are obtained by using the number of frames rendered per unit of time, the rendering latency rate, and the memory utilization rate.
[0113] The rendering operation coefficients are obtained using the following formula:
[0114]
[0115] Where R represents the rendering runtime coefficient; Z represents the number of frames rendered per unit time; M represents the memory utilization rate rendered per unit time; and D represents the rendering latency rate rendered per unit time.
[0116] The rendering operation coefficients are compared with preset coefficient thresholds;
[0117] When the rendering operation coefficient exceeds the preset coefficient threshold, the rendering operation coefficient corresponding to all unit time is retrieved to obtain the comprehensive coefficient value.
[0118] The comprehensive coefficient value is obtained by the following formula:
[0119]
[0120] Where S represents the comprehensive coefficient value; n represents the number of unit time periods; R i R represents the rendering runtime coefficient corresponding to the i-th unit of time; c R represents the rendering performance coefficient that exceeds a preset threshold. pR represents the average rendering performance coefficient over n units of time; b R represents the standard deviation of the rendering runtime coefficients over n units of time; y This indicates a preset coefficient threshold.
[0121] The comprehensive coefficient value is compared with a preset comprehensive coefficient threshold.
[0122] When the composite coefficient value exceeds the preset composite coefficient threshold, it is determined that there is an abnormality in the rendering process.
[0123] The working principle of the above technical solution is as follows: Real-time acquisition of core rendering operation parameters, including the number of rendering frames (Z), rendering latency (D), and memory utilization (M) per unit time, as basic data for evaluating the rendering status; Calculation of the rendering operation coefficient (R) per unit time based on the above parameters using a specific formula, integrating multi-dimensional parameters into a single quantitative indicator to reflect the rendering operation status within that time unit; Comparison of the rendering operation coefficient (R) per unit time with a preset threshold for preliminary anomaly screening, triggering further evaluation only when R exceeds the threshold; For scenarios that trigger preliminary screening, historical rendering operation coefficients per unit time are retrieved, and a comprehensive coefficient value (S) is calculated using a comprehensive formula—this value integrates the coefficient exceeding the threshold (Rc), the historical average coefficient (Rp), the historical standard deviation (Rb), and the preset threshold (Ry), comprehensively evaluating the stability of the rendering status from the perspective of statistical distribution and deviation; Finally, by comparing the comprehensive coefficient value (S) with the preset comprehensive threshold, it is determined whether there are any anomalies in the rendering operation process.
[0124] The effects of the above technical solution are as follows: Real-time acquisition and calculation of core rendering parameters enable dynamic tracking of the rendering state, ensuring timely detection of any anomalies; integration of dispersed parameters such as frame rate, latency, and memory utilization into a rendering operation coefficient (R) allows for single-dimensional quantitative evaluation of the rendering state, simplifying the complexity of state judgment; the two-layer logic of "single-unit time coefficient threshold screening → multi-unit time comprehensive coefficient evaluation" reduces misjudgments caused by fluctuations at a single time point, improving the rigor of anomaly judgment; the comprehensive coefficient (S) incorporates statistical features such as historical mean and standard deviation, taking into account both the overall trend and short-term fluctuations of the rendering state, making anomaly judgment more consistent with actual operating patterns; and through multi-layer verification and multi-factor fusion, it effectively distinguishes between normal fluctuations and substantial anomalies, providing a reliable basis for subsequent optimization adjustments (such as resource allocation and parameter adaptation), ensuring the stability and efficiency of the JPEG encoding process in the 3D engine.
[0125] This invention proposes a system for rapidly encoding images in a 3D engine, such as... Figure 2 As shown, the system for rapidly encoding images in a 3D engine includes:
[0126] The rendering position differentiation module is used to distinguish the approach of rendering each camera onto a separate RenderTexture. This invention renders the content of multiple cameras onto the same RenderTexture and differentiates the positions of different cameras:
[0127] The graphics library pointer acquisition module is used to obtain the underlying graphics library pointer of the RenderTexture object through the RenderTexture.GetNativeTexturePtr method provided by Unity.
[0128] The resource format conversion control module is used to obtain the graphics pointer through RenderTexture, convert the resource format and map it to CUDA memory, use GPU encoding to the preset format, and reuse video memory to reduce resource consumption.
[0129] The working principle and technical effect of the above technical solution are as follows: The underlying graphics library pointer of the RenderTexture object is obtained through the RenderTexture.GetNativeTexturePtr method provided by Unity. For this invention, assuming that Unity uses DX11 as its graphics library, the ID3D11Texture2D object is obtained. By obtaining the original graphics pointer, this invention uses the methods provided by DX11 to convert the graphics resources in Unity into resources that can be used subsequently. During this process, the original resource format (DXGI_FORMAT_R8G8B8A8_TYPELESS) is converted into a subsequently usable format (DXGI_FORMAT_R8G8B8A8_UNORM). The obtained ID3D11Texture2D resource is then mapped to a two-dimensional memory resource cudaArray in CUDA using CUDA methods (cudaGraphicsD3D11RegisterResource, cudaGraphicsMapResources). Using the cudaMemcpy2DFromArray method, this invention converts cudaArray resources into uint8_t type video memory pointers and inputs them into a GPU-based JPEG encoding algorithm. The final output is a pointer to JPEG byte data encoded from the original pixel data. During the process, various resources are validated based on the JPEG image width and height. If the previous width and height input are the same as the current input, the original video memory resources are reused to reduce consumption. The above code is written in C++, CUDA, and DX11 environments, compiled into a DLL, and used in Unity's C# implementation. The C# Marshal library is used to convert C pointers into C# byte arrays.
[0130] In one embodiment of the present invention, the rendering position differentiation module includes:
[0131] The rendering control module is used to render the content of multiple cameras onto the same RenderTexture, and defines an integer variable i, initialized to 0, for counting and indexing;
[0132] The rendering settings module is used to iterate through an array or collection of cameras named cameras using a foreach loop and set the rendering settings for each camera.
[0133] The region creation module is used to create rectangular regions and increment the value of the counter i so that the next camera can be processed in the next loop.
[0134] The working principle of the above technical solution is as follows: This embodiment uses a single RenderTexture as a unified rendering target, and integrates and distinguishes the content of multiple cameras through the following logic: Define an integer variable i and initialize it to 0 to mark the index of the currently processed camera and calculate its corresponding rendering area; when traversing the cameras array, configure the rendering target of each camera as the same RenderTexture; dynamically calculate the rectangular area (such as position and size) of the camera on the RenderTexture based on the value of i, to ensure that the rendering content of each camera is limited to an independent area; after each loop, i is incremented to allocate a new area for the next camera and avoid content overlap.
[0135] The effects of the above technical solution are as follows: it integrates the camera content that originally required multiple rendering targets into a single RenderTexture, reducing resource consumption (such as memory and video memory consumption); through the rectangular area division controlled by i, it ensures that the content of each camera maintains spatial independence on the RenderTexture, forming a structured combined display; the unified rendering target reduces the overhead of rendering state switching, and subsequent processing of multi-camera content (such as post-processing and display) can be completed based on a single RenderTexture, simplifying the process; the rendering logic of each camera is not affected by other cameras, and its original content is completely preserved in the specified area, ensuring that no information is lost.
[0136] In one embodiment of the present invention, the rendering settings module includes:
[0137] The label checking module checks whether the label of each camera is "UnitCamera"; if the label does not match, the current loop iteration is skipped and the process continues to the next camera.
[0138] The attribute setting module is used to set the targetTexture attribute of the current camera to camRenderTex, where camRenderTex is a rendering target texture;
[0139] The path setting module is used to set the current camera's rendering path to RenderingPath.Forward, instructing the camera to use forward rendering;
[0140] The background fill module is used to set the camera's clear flag to CameraClearFlags.SolidColor, and fill the background with a single color for each frame rendered.
[0141] The background color setting module is used to set the camera's background color to white (Color.white);
[0142] The shader setting module is used to call the camera's SetReplacementShader method to set the effects shader.
[0143] The child object lookup module is used to find the child object "UnityDrone" of the current camera object and set it to inactive (i.e. disable the game object).
[0144] The row and column position determination module is used to calculate the modulus (remainder) of the current index i divided by the total number of columns totalCol, and assign the result to the variable col. It also calculates the result of dividing the current index i by the total number of columns totalCol and assigns it to the variable row. This will be used to determine the row and column positions of the camera in the grid.
[0145] The region creation module includes:
[0146] The rectangle object creation module is used to create a new rectangle object cam.rect using the Rect struct, defining its position and size:
[0147] The X-coordinate setting module is used to set the X-coordinate of the top left corner as col * 1.0 / totalCol, which represents the proportion occupied in the column;
[0148] The Y-coordinate setting module is used to set the Y-coordinate of the top left corner as row * 1.0 / totalRow, representing the proportion it occupies in the row.
[0149] The width and height setting module is used to set the width and height to 1.0 / totalCol and 1.0 / totalRow, respectively, representing the proportion of each camera in the total columns and total rows.
[0150] The working principle of the above technical solution is as follows: First, cameras are filtered by the label "UnitCamera", and only cameras that meet the conditions are processed, while irrelevant objects are filtered out; the filtered cameras are uniformly configured with rendering targets (camRenderTex), rendering paths (forward rendering), cleanup rules (solid color background fill), and background color (white) to ensure that the basic rendering properties are consistent; a unified effect shader is applied through SetReplacementShader to make the content of each camera present a unified style; the child object "UnityDrone" is disabled to exclude specific elements that do not need to be rendered, simplifying the screen; the row and column positions of each camera in the grid are determined based on the calculation of the current index i and the total number of columns totalCol and the total number of rows totalRow (col = i % totalCol, row = i / totalCol); a rectangular area is defined with proportional coordinates (col / totalCol, row / totalRow are the top left corners, 1 / totalCol, 1 / totalRow are the width and height), which limits the rendering content of each camera to the corresponding grid cell.
[0151] The effects of the above technical solution are as follows: Tag filtering processes only the target camera, reducing unnecessary calculations and improving rendering efficiency; unified rendering targets, paths, cleanup rules, and background colors ensure consistency in the underlying rendering logic of each camera's content, avoiding visual conflicts; the application of shader replacement ensures that all camera content presents the same special effects style, enhancing the overall picture coherence; disabling specific sub-objects and eliminating redundant elements allows the rendered content to focus more on core information; proportional rectangular areas based on row and column calculations enable the content of multiple cameras to form a regular grid arrangement on the RenderTexture, avoiding content overlap and ensuring layout adaptation (unaffected by the actual resolution of the RenderTexture), achieving orderly and compact multi-screen integration; unified target textures reduce rendering state switching overhead, providing convenience for subsequent secondary processing of the integrated picture (such as display and post-processing).
[0152] In one embodiment of the present invention, the resource format conversion control module includes:
[0153] The resource format conversion module is used to convert graphics resources in Unity into a resource format that meets preset requirements by obtaining the pointer to the underlying graphics library and using the methods provided by DX11; specifically, it converts the original resource format (DXGI_FORMAT_R8G8B8A8_TYPELESS) into a format that can be used later (DXGI_FORMAT_R8G8B8A8_UNORM).
[0154] The mapping control module is used to map the converted resource format to a two-dimensional memory resource cudaArray in CUDA using CUAD methods (cudaGraphicsD3D11RegisterResource, cudaGraphicsMapResources);
[0155] The image output module is used to convert the two-dimensional memory resource cudaArray into a uint8_t* type video memory pointer and input it to the GPU-based JPEG encoding algorithm. The final output is a JPEG byte data pointer that encodes the original pixel data into JPEG. The module also verifies the width and height of the JPEG image. When the width and height of the current JPEG image input are the same as the previous JPEG image input, the original video memory resources are reused to reduce consumption.
[0156] The working principle of the above technical solution is as follows: Obtain the underlying graphics library pointer from RenderTexture; convert the original typeless format (DXGI_FORMAT_R8G8B8A8_TYPELESS) into a standardized unsigned normalized format (DXGI_FORMAT_R8G8B8A8_UNORM) based on the DX11 interface to ensure that the resource format meets the compatibility requirements of subsequent processing; utilize CUDA's graphics resource interaction methods (cudaGraphicsD3D11RegisterResource, cudaGraphicsMapResources) to map the converted graphics resources into CUDA-accessible two-dimensional memory resources (cudaArray), establishing a direct association between CPU-side graphics resources and GPU-side computing resources; convert the cudaArray into a uint8_t * type memory pointer, and pass it as input to the GPU-implemented JPEG encoding algorithm to complete the encoding conversion from pixel data to JPEG byte data; by verifying the consistency of the current JPEG image's width and height with the previous one, reuse the allocated memory resources when the conditions are met, avoiding repeated memory allocation and release operations.
[0157] The effects of the above technical solutions are as follows: format conversion ensures that graphics resources meet the format requirements of subsequent processing (such as CUDA access and encoding algorithms), eliminating processing anomalies caused by format incompatibility; CUDA mapping enables direct access of graphics resources to GPU memory, reducing data copy overhead between the CPU and GPU, and combined with GPU encoding algorithms, significantly improving encoding efficiency and breaking through the performance bottleneck of CPU encoding; the memory reuse mechanism avoids repeated memory allocation in scenes with the same width and height, reduces memory fragmentation and memory bandwidth occupation, and reduces latency caused by resource allocation and release; the entire process from graphics resource acquisition to JPEG encoding relies on GPU acceleration and resource reuse, shortening the overall processing cycle and enhancing the real-time performance and stability of the system in high frame rate and high resolution scenes.
[0158] In one embodiment of the present invention, the method for rapidly encoding images in a 3D engine further includes:
[0159] Real-time monitoring of the current camera's rendering operation parameters, including the number of frames rendered per unit time, rendering latency, and memory utilization.
[0160] The rendering operation coefficients corresponding to each unit of time are obtained by using the number of frames rendered per unit of time, the rendering latency rate, and the memory utilization rate.
[0161] The rendering operation coefficients are obtained using the following formula:
[0162]
[0163] Where R represents the rendering runtime coefficient; Z represents the number of frames rendered per unit time; M represents the memory utilization rate rendered per unit time; and D represents the rendering latency rate rendered per unit time.
[0164] The rendering operation coefficients are compared with preset coefficient thresholds;
[0165] When the rendering operation coefficient exceeds the preset coefficient threshold, the rendering operation coefficient corresponding to all unit time is retrieved to obtain the comprehensive coefficient value.
[0166] The comprehensive coefficient value is obtained by the following formula:
[0167]
[0168] Where S represents the comprehensive coefficient value; n represents the number of unit time periods; R i R represents the rendering runtime coefficient corresponding to the i-th unit of time; c R represents the rendering performance coefficient that exceeds a preset threshold. pR represents the average rendering performance coefficient over n units of time; b R represents the standard deviation of the rendering runtime coefficients over n units of time; y This indicates a preset coefficient threshold.
[0169] The comprehensive coefficient value is compared with a preset comprehensive coefficient threshold.
[0170] When the composite coefficient value exceeds the preset composite coefficient threshold, it is determined that there is an abnormality in the rendering process.
[0171] The working principle of the above technical solution is as follows: Real-time acquisition of core rendering operation parameters, including the number of rendering frames (Z), rendering latency (D), and memory utilization (M) per unit time, as basic data for evaluating the rendering status; Calculation of the rendering operation coefficient (R) per unit time based on the above parameters using a specific formula, integrating multi-dimensional parameters into a single quantitative indicator to reflect the rendering operation status within that time unit; Comparison of the rendering operation coefficient (R) per unit time with a preset threshold for preliminary anomaly screening, triggering further evaluation only when R exceeds the threshold; For scenarios that trigger preliminary screening, historical rendering operation coefficients per unit time are retrieved, and a comprehensive coefficient value (S) is calculated using a comprehensive formula—this value integrates the coefficient exceeding the threshold (Rc), the historical average coefficient (Rp), the historical standard deviation (Rb), and the preset threshold (Ry), comprehensively evaluating the stability of the rendering status from the perspective of statistical distribution and deviation; Finally, by comparing the comprehensive coefficient value (S) with the preset comprehensive threshold, it is determined whether there are any anomalies in the rendering operation process.
[0172] The effects of the above technical solution are as follows: Real-time acquisition and calculation of core rendering parameters enable dynamic tracking of the rendering state, ensuring timely detection of any anomalies; integration of dispersed parameters such as frame rate, latency, and memory utilization into a rendering operation coefficient (R) allows for single-dimensional quantitative evaluation of the rendering state, simplifying the complexity of state judgment; the two-layer logic of "single-unit time coefficient threshold screening → multi-unit time comprehensive coefficient evaluation" reduces misjudgments caused by fluctuations at a single time point, improving the rigor of anomaly judgment; the comprehensive coefficient (S) incorporates statistical features such as historical mean and standard deviation, taking into account both the overall trend and short-term fluctuations of the rendering state, making anomaly judgment more consistent with actual operating patterns; and through multi-layer verification and multi-factor fusion, it effectively distinguishes between normal fluctuations and substantial anomalies, providing a reliable basis for subsequent optimization adjustments (such as resource allocation and parameter adaptation), ensuring the stability and efficiency of the JPEG encoding process in the 3D engine.
[0173] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.
Claims
1. A method for rapidly encoding images in a 3D engine, characterized in that, The method for rapidly encoding images in a 3D engine includes: Render the content of multiple cameras onto the same RenderTexture, and differentiate the positions of different cameras; The underlying graphics library pointer of the RenderTexture object can be obtained through the RenderTexture.GetNativeTexturePtr method provided by Unity; The graphics pointer is obtained through RenderTexture, the resource format is converted and mapped to CUDA memory, and the GPU is used to encode it into a preset format, reusing video memory to reduce resource consumption.
2. The method for rapidly encoding images in a 3D engine according to claim 1, characterized in that, Render the content of multiple cameras onto the same RenderTexture, and differentiate the positions of different cameras, including: Render the content of multiple cameras onto the same RenderTexture, and define an integer variable i, initialized to 0, for counting and indexing; Use a foreach loop to iterate through the array or collection of cameras named cameras and set the rendering for each camera; Create a rectangular region and increment the value of the counter i so that the next camera can be processed in the next loop.
3. The method for rapidly encoding images in a 3D engine according to claim 2, characterized in that, Render settings for each camera, including: For each camera, check if its label is "UnitCamera"; if the label does not match, skip the current loop iteration and continue to the next camera. Set the targetTexture property of the current camera to camRenderTex, where camRenderTex is a rendering target texture; Set the current camera's rendering path to RenderingPath.Forward, instructing the camera to use forward rendering; Set the camera's clear flag to CameraClearFlags.SolidColor, and fill the background with a single color for each frame rendered. Set the camera's background color to white; Call the camera's SetReplacementShader method to set the effect shader; Find the current camera object's child object "UnityDrone" and set it to inactive; Calculate the modulus of the current index i with respect to the total number of columns totalCol, and assign the result to the variable col. Also, calculate the result of dividing the current index i by the total number of columns totalCol, and assign it to the variable row.
4. The method for rapidly encoding images in a 3D engine according to claim 2, characterized in that, Creating a rectangular region includes: Create a new rectangle object cam.rect using the Rect struct, defining its position and size: The X-coordinate of the top-left corner is col * 1.0 / totalCol, representing the proportion it occupies in the column; The Y-coordinate of the top-left corner is row * 1.0 / totalRow, representing the proportion it occupies in the row; The width and height are both 1.0 / totalCol and 1.0 / totalRow, representing the proportion of each camera in the total columns and total rows.
5. The method for rapidly encoding images in a 3D engine according to claim 1, characterized in that, Obtain the graphics pointer through RenderTexture, convert the resource format and map it to CUDA memory, encode it into a preset format using the GPU, and reuse video memory to reduce resource consumption, including: By obtaining the pointer to the underlying graphics library, the graphics resources in Unity are converted into a resource format that meets the preset requirements; The converted resource format is mapped to a two-dimensional memory resource cudaArray in CUDA using the CUAD method; The two-dimensional memory resource cudaArray is converted into a uint8_t* type video memory pointer and input to the GPU-based JPEG encoding algorithm. The final output is a pointer to JPEG byte data that encodes the original pixel data into JPEG. The width and height of the JPEG image are verified. When the width and height of the current JPEG image input are the same as the previous JPEG image input, the original video memory resources are reused to reduce consumption.
6. A system for rapidly encoding images in a 3D engine, characterized in that, The system for rapidly encoding images in a 3D engine includes: The rendering position differentiation module is used to render the content of multiple cameras onto the same RenderTexture and differentiate the positions of different cameras; The graphics library pointer acquisition module is used to obtain the underlying graphics library pointer of the RenderTexture object through the RenderTexture.GetNativeTexturePtr method provided by Unity. The resource format conversion control module is used to obtain the graphics pointer through RenderTexture, convert the resource format and map it to CUDA memory, use GPU encoding to the preset format, and reuse video memory to reduce resource consumption.
7. The system for rapidly encoding images in a 3D engine according to claim 6, characterized in that, The rendering position differentiation module includes: The rendering control module is used to render the content of multiple cameras onto the same RenderTexture, and defines an integer variable i, initialized to 0, for counting and indexing; The rendering settings module is used to iterate through an array or collection of cameras named cameras using a foreach loop and set the rendering settings for each camera. The region creation module is used to create rectangular regions and increment the value of the counter i so that the next camera can be processed in the next loop.
8. The system for rapidly encoding images in a 3D engine according to claim 7, characterized in that, The rendering settings module includes: The label checking module checks whether the label of each camera is "UnitCamera"; if the label does not match, the current loop iteration is skipped and the process continues to the next camera. The attribute setting module is used to set the targetTexture attribute of the current camera to camRenderTex, where camRenderTex is a rendering target texture; The path setting module is used to set the current camera's rendering path to RenderingPath.Forward, instructing the camera to use forward rendering; The background fill module is used to set the camera's clear flag to CameraClearFlags.SolidColor, and fill the background with a single color for each frame rendered. The background color setting module is used to set the camera's background color to white; The shader setting module is used to call the camera's SetReplacementShader method to set the effect shader; The child object lookup module is used to find the child object "UnityDrone" of the current camera object and set it to inactive; The row and column position determination module is used to calculate the modulus of the current index i with respect to the total number of columns totalCol, assign the result to the variable col, and calculate the result of dividing the current index i by the total number of columns totalCol, assigning it to the variable row.
9. The system for rapidly encoding images in a 3D engine according to claim 7, characterized in that, The region creation module includes: The rectangle object creation module is used to create a new rectangle object cam.rect using the Rect struct, defining its position and size: The X-coordinate setting module is used to set the X-coordinate of the top left corner as col * 1.0 / totalCol, which represents the proportion occupied in the column; The Y-coordinate setting module is used to set the Y-coordinate of the top left corner as row * 1.0 / totalRow, which represents the proportion of the row. The width and height setting module is used to set the width and height to 1.0 / totalCol and 1.0 / totalRow, respectively, representing the proportion of each camera in the total columns and total rows.
10. The system for rapidly encoding images in a 3D engine according to claim 6, characterized in that, The resource format conversion control module includes: The resource format conversion module is used to convert graphics resources in Unity into a resource format that meets preset requirements by obtaining the pointer to the underlying graphics library; The mapping control module is used to map the converted resource format to a two-dimensional memory resource cudaArray in CUDA using the CUAD method; The image output module is used to convert the two-dimensional memory resource cudaArray into a uint8_t* type video memory pointer and input it to the GPU-based JPEG encoding algorithm. The final output is a JPEG byte data pointer that encodes the original pixel data into JPEG. The module also verifies the width and height of the JPEG image. When the width and height of the current JPEG image input are the same as the previous JPEG image input, the original video memory resources are reused to reduce consumption.