System for automatically diagnosing rendering exception based on ue5 pixel readback and opencv

By constructing an automated rendering anomaly diagnosis system based on UE5 pixel readback and OpenCV, the problem of limited functionality and low detection rate of existing rendering anomaly diagnosis tools is solved. It realizes multimodal detection and real-time diagnosis of complex rendering anomalies, can accurately locate the cause of anomalies, reduce troubleshooting time, and supports online detection and cross-frame analysis.

CN120823211BActive Publication Date: 2025-12-12SHANGHAI YINSHENG INFORMATION TECH CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202511331816.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2025-09-18
Publication Date
2025-12-12
Estimated Expiration
2045-09-18

AI Technical Summary

Technical Problem

In existing technical solutions, in the traditional field of computer graphics, in fields such as virtual reality (VR/AR) and building information modeling (BIM), in complex rendering debugging, the existing technologies have the following technical problems: In the field of computer graphics, the existing rendering anomaly diagnosis tools have limited functions and low detection rates, and cannot efficiently solve the problems of multi-factor coupling, non-deterministic performance and long debugging cycles.

Method used

By constructing an automated diagnostic system for rendering anomalies based on UE5 pixel readback and OpenCV, including a pipeline extension layer, a data capture layer, and an intelligent analysis layer, rendering data is collected using a geometry marker pass, a performance probe pass, and a feature enhancement pass. Multi-dimensional anomaly detection is performed using OpenCV to achieve intelligent diagnosis of rendering anomalies.

Benefits of technology

It achieves multimodal detection of complex rendering anomalies, has real-time diagnostic capabilities, and can pinpoint the source of anomalies at the model/material/shader granularity, reducing troubleshooting time and improving positioning accuracy. It supports online detection and cross-frame analysis, covering anomaly types such as performance bottlenecks, visual defects, and rendering logic errors.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN120823211B_ABST
    Figure CN120823211B_ABST
Patent Text Reader

Abstract

The application discloses a rendering abnormality automatic diagnosis system based on UE5 pixel readback and OpenCV, which comprises a pipeline extension layer, a data capture layer and an intelligent analysis layer; wherein the pipeline extension layer constructs a rendering pipeline extension model and collects RenderTarget data; the data capture layer realizes asynchronous reading of the RenderTarget by adopting a double buffering mechanism; the intelligent analysis layer constructs a multi-dimensional abnormality detection model based on OpenCV and analyzes image data to obtain abnormality analysis results. The scheme realizes multi-modal abnormality detection diagnosis through the construction of a three-layer architecture of "rendering pipeline extension-intermediate result capture-intelligent analysis decision", and can cover nine categories of abnormal types such as performance bottleneck, visual defect and rendering logic error. The scheme realizes intelligent diagnosis of rendering defects by inserting a customized Debug rendering Pass, capturing rendering intermediate results in real time and combining computer vision algorithms.
Need to check novelty before this filing date? Find Prior Art

Description

TECHNICAL FIELD

[0001] The application belongs to the field of computer graphics, and particularly relates to a rendering exception automatic diagnosis system based on UE5 pixel readback and OpenCV. BACKGROUND

[0002] In the fields of digital content creation (DCC), virtual reality (VR / AR), building information modeling (BIM), etc., the debugging of complex rendering pipelines faces significant challenges: the traditional debugging method relying on manual frame capture has three major pain points:

[0003] 1. Multi-factor coupling: rendering exceptions can be caused by the interaction of multiple links such as Shader code, material parameters, geometric topology, and lighting settings.

[0004] 2. Non-deterministic behavior: some abnormal phenomena (such as Z-fighting) have randomness and time sequence correlation.

[0005] 3. Long debugging period: the full-process debugging of a typical project can take up to 8-12 hours.

[0006] The existing technical solutions have obvious limitations, mainly as follows:

[0007] 1. The built-in diagnostic tools of commercial engines have limited functions (such as UE5 Frame Debugger which only supports single-frame static analysis).

[0008] 2. Third-party performance analysis software (such as PIX, RenderDoc) lacks intelligent exception identification capabilities.

[0009] 3. Traditional algorithms based on rule matching have an insufficient detection rate of 43% for complex abnormal patterns. SUMMARY

[0010] The technical problem to be solved by the application is to provide a rendering exception automatic diagnosis system based on UE5 pixel readback and OpenCV, which solves the problem of low detection rate and single function of the traditional diagnosis system in the prior art.

[0011] The application adopts the following technical solutions to solve the above technical problems:

[0012] The rendering exception automatic diagnosis system based on UE5 pixel readback and OpenCV comprises a pipeline extension layer, a data capture layer, and an intelligent analysis layer; the pipeline extension layer constructs a rendering pipeline extension model and collects RenderTarget data; the data capture layer realizes asynchronous reading of RenderTarget by using a double buffering mechanism; the intelligent analysis layer constructs a multi-dimensional exception detection model based on OpenCV and analyzes image data to obtain an exception analysis result.

[0013] The rendering pipeline extension model includes three types of customized Pass implanted in the UE5 standard rendering process, which are:

[0014] A geometry marking Pass for recording the RGBA encoding information of model instance ID / material ID;

[0015] A performance probe Pass for inserting GPU timestamp markers in key rendering stages;

[0016] A feature enhancement Pass for visual feature enhancement of potential abnormal areas.

[0017] The specific process of collecting RenderTarget data is as follows:

[0018] First, add a custom DebugPass after a specific rendering Pass according to different requirements; second, collect the RenderTarget of the custom and built-in DebugPass and normal rendering Pass.

[0019] The data capture layer reads the collected data in the RenderTarget and saves it to a UTexture2D object; and uses the data in the UTexture2D to create a corresponding cv::mat object.

[0020] The process of analyzing image data includes:

[0021] Scene color anomaly detection before post-processing, dramatic changes between consecutive frames at the same position, screen tearing detection, semi-transparent rendering order anomaly detection, and rendering artifact detection.

[0022] Dramatic changes between consecutive frames at the same position include screen flicker and moire, which are detected using a combination of inter-frame difference method and optical flow method.

[0023] Rendering artifact detection includes jaggies and moire, which are detected using edge detection and frequency analysis methods to detect the sharpness of edges and detect artifacts through frequency domain transformation.

[0024] It also includes a performance anomaly analysis module, and the specific analysis process is as follows:

[0025] Insert GPU time query instructions between each DrawCall to obtain the execution time of each DrawCall, and convert the data into color and write it into the corresponding pixel. Accumulate in each DrawCall, finally observe the generated image to determine the pixels that consume too much, and through the pre-generated pixel correspondence with models and materials, obtain the materials or models that may cause performance anomalies.

[0026] Compared with the prior art, the present application has the following beneficial effects:

[0027] 1. The present application realizes multi-modal anomaly detection and diagnosis through the construction of a three-layer architecture of "rendering pipeline extension-intermediate result capture-smart analysis and decision", which can cover 9 categories of abnormal types such as performance bottleneck, visual defect, rendering logic error, etc.

[0028] 2. The present application realizes intelligent diagnosis of rendering defects by inserting customized Debug rendering Pass, capturing rendering intermediate results in real time, and combining computer vision algorithms.

[0029] 3. The present application can realize spatio-temporal correlation analysis: dynamic anomaly tracking is realized through inter-frame difference + optical flow tracking; root cause positioning accuracy: anomaly tracing at the granularity of model / material / Shader is realized.

[0030] 4. The system has real-time diagnosis capability: online detection at 120fps frame rate is supported.

[0031] 5. The present application can skip the originally complex manual debugging steps (anomaly reproduction-capture single-frame rendering data-analyze rendering data) and automatically analyze the entire rendering process.

[0032] 6. Cross-frame anomaly analysis capability: the original analysis process can only analyze single-frame data, and if the anomaly is not caused by the current frame, it is difficult to locate. The present application can locate the anomaly source caused by non-current frames through continuous multi-frame data analysis, such as material flicker or ghosting.

[0033] 7. High positioning accuracy: the method of marking models and materials by pixels improves the anomaly cause positioning accuracy to a single material or model, reducing the troubleshooting time. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 The present application is based on UE5 pixel readback and OpenCV rendering anomaly automatic diagnosis system architecture.

[0035] Figure 2 The present application is based on UE5 pixel readback and OpenCV rendering anomaly automatic diagnosis effect comparison chart. DETAILED DESCRIPTION

[0036] The structure and working process of the present application will be further described below in conjunction with the drawings.

[0037] This solution achieves anomaly diagnosis by constructing a three-layer architecture of "rendering pipeline extension - intermediate result capture - intelligent analysis and decision-making". Specifically, the automated rendering anomaly diagnosis system based on UE5 pixel readback and OpenCV (image analysis library) includes a pipeline extension layer, a data capture layer, and an intelligent analysis layer. Among them, the pipeline extension layer constructs a rendering pipeline extension model and collects RenderTarget (rendering target, an image used to store rendering data) data; the data capture layer uses a double buffering mechanism to achieve asynchronous reading of RenderTarget; the intelligent analysis layer constructs a multi-dimensional anomaly detection model based on OpenCV and analyzes image data to obtain anomaly analysis results.

[0038] Specific embodiments, such as Figure 1 , Figure 2 As shown:

[0039] An automated rendering anomaly diagnosis system based on UE5 pixel readback and OpenCV includes a pipeline extension layer, a data capture layer, and an intelligent analysis layer; among which,

[0040] 1. Pipeline Extension Layer: Incorporating three types of customized passes into the UE5 standard rendering pipeline:

[0041] Geometric Marker Pass: Records the RGBA encoding information of the model instance ID / material ID. The specific operation process is as follows:

[0042] Insert a custom geometry markup Pass into the PostBasePass of the UE5 rendering pipeline. This Pass iterates through all currently rendered model instances, generates a unique ID code for each instance (e.g., model ID is 0x00FF, material ID is 0x0F00), and converts it to RGBA color values ​​(the lower 8 bits of the model ID are mapped to the R channel, and the higher 8 bits are mapped to the G channel; the lower 8 bits of the material ID are mapped to the B channel, and the highest bit is reserved and mapped to the A channel).

[0043] The encoding rules need to be predefined (e.g., ID = (ModelID<<16)|(MaterialID<<8), corresponding to R=ModelID&0xFF, G=(ModelID>>8)&0xFF, B=MaterialID&0xFF,A=0xFF).

[0044] Example:

[0045] Assuming the model ID is 1024 (hexadecimal 0x0400) and the material ID is 255 (hexadecimal 0x00FF), then the encoded RGBA value is:

[0046] R = 1024 & 0xFF = 0x00

[0047] G = (1024 >> 8) & 0xFF = 0x04,

[0048] B = 255 & 0xFF = 0xFF,

[0049] A = 0xFF (opaque) Finally, all pixels of this model instance will be marked as (0x00, 0x04, 0xFF, 0xFF) (blue).

[0050] Performance Probe Pass: Insert GPU timestamp markers in critical rendering stages. The specific operation process is as follows:

[0051] Insert performance probe Pass in critical rendering stages (such as BasePass, ShadowMapPass, PostProcessPass), each probe inserts a timestamp marker into the GPU timeline by calling RHICmdList.WriteTimestamp(), recording the start / end GPU time of the stage. Timestamp data is stored in a pre-allocated GPU timestamp buffer (size N x 8 bytes, N is the maximum number of probes).

[0052] Example:

[0053] Assume that probe 1 is inserted at the beginning of BasePass and probe 2 is inserted at the end; probe 3 is inserted at the beginning of PostProcessPass; the timestamp sequence recorded by the GPU timeline is [T0, T1, T2, T3], where:

[0054] T0: BasePass start time,

[0055] T1: BasePass end time → BasePass duration = T1 - T0,

[0056] T2: PostProcessPass start time,

[0057] T3: PostProcessPass end time → PostProcessPass duration = T3 - T2.

[0058] Data association: Bind timestamps with specific DrawCall (each DrawCall corresponds to the rendering instructions of a model instance) through UE5's DrawCall index, and locate the long-time-consuming DrawCall and its associated model / material in the future.

[0059] Feature Enhancement Pass: Visual feature enhancement is performed on potentially anomalous areas (such as highlights and shadows). The specific operation process is as follows:

[0060] For areas prone to anomalies (such as highlights, shadows, and transparent edges), a feature enhancement pass is inserted into the rendering pipeline. By adjusting the shader logic, the color values ​​of these areas are enhanced.

[0061] Highlight areas: Extract the Z component of the normal map (representing the surface orientation). If the angle between the normal and the viewpoint is less than 15° (i.e. close to the specular reflection direction), increase the brightness of the area by 30% (Color.rgb *= 1.3).

[0062] Shadow areas: Extract the gradient of the depth map (DepthGradient = dDepth / dx + dDepth / dy). If the gradient is greater than the threshold (indicating the edge of the shadow), the contrast of the area will be increased by 20% ((Color - 0.5) * 1.2 + 0.5).

[0063] Transparent edges: Extract the gradient of the Alpha channel (AlphaGradient = dAlpha / dx + dAlpha / dy). If the gradient is greater than the threshold (indicating a semi-transparent edge), increase the saturation of the area by 40% (Color.rgb = saturate(Color.rgb * 1.4)).

[0064] Example:

[0065] The metal helmet of a character model produces highlights under illumination, with the original highlight area brightness being (1.0, 0.9, 0.8). After feature enhancement processing, the brightness is increased by 30%, becoming (1.3, 1.17, 1.04) (due to HDR range limitations, it will actually be truncated to (1.0, 1.0, 1.0)). However, the outline of the highlight area is more easily recognized by subsequent analysis modules due to the increased brightness.

[0066] 2. Data Capture Layer: A double-buffering mechanism is used to implement asynchronous reading of RenderTarget. The specific implementation process is as follows:

[0067] First, get the RenderTarget (back-end buffer) of the SceneCaptureComponent, submit an asynchronous read task to the GPU command queue; get the RHI texture resource (texture in GPU memory) of the RenderTarget; then, perform a pixel data read operation, create a temporary CPU-readable texture (confirm the memory size by RHIGetTextureMemorySize), copy the contents of the RenderTarget to the temporary texture (copy within the GPU, without blocking the CPU); finally, map the temporary texture data to the CPU memory (lock the texture, get the pointer).

[0068] Collect the RenderTarget from custom and UE built-in DebugPass and normal rendering Pass, this collection process will last for multiple frames to detect abnormal changes between consecutive frames.

[0069] C++ example code as follows:

[0070] / / UE5 C++ example code

[0071] Get the RenderTarget (back-end buffer) of the SceneCaptureComponent

[0072] TSharedPtr <frendertarget>RenderTarget =

[0073] SceneCaptureComponent->TextureTarget;

[0074] Submit the asynchronous read task to the GPU command queue

[0075] ENQUEUE_RENDER_COMMAND(CaptureRenderTargetCommand)(

[0076] [RenderTarget](FRHICommandListImmediate& RHICmdList)

[0077] {

[0078] Get the RHI texture resource (texture in GPU memory) of the RenderTarget

[0079] FTexture2DRHIRef TextureRHI = RenderTarget->GetRenderTargetResource()->GetRenderTargetTexture();

[0080] / / Perform a pixel data read operation

[0081] Create a temporary CPU-readable texture (confirm memory size by RHIGetTextureMemorySize)

[0082] FTexture2DRHIRef ReadbackTexture = RHICreateTexture2D();

[0083] Copy the contents of the RenderTarget to the temporary texture (copy in GPU, not blocking CPU)

[0084] RHICmdList.CopyToResolveTarget(TextureRHI, ReadbackTexture,FResolveParams());

[0085] Map the temporary texture data to CPU memory (lock the texture, get the pointer)

[0086] void* DataPtr;

[0087] uint32 Pitch;

[0088] RHICmdList.LockTexture(ReadbackTexture->TextureRHI, 0, FRHIGPUFence(), false);

[0089] FTexture2DRHI::GetRenderTargetData(ReadbackTexture, DataPtr, Pitch);

[0090] });

[0091] 3. Intelligent analysis layer: build an OpenCV-based multi-dimensional anomaly detection model that analyzes image data. The specific analysis process and implementation code are as follows:

[0092] # OpenCV core processing flow (Python pseudo code)

[0093] def analyze_frame(mat_frame):

[0094] # Performance anomaly detection module

[0095] drawcall_heatmap = generate_drawcall_heatmap(mat_frame)

[0096] Function:

[0097] Map the DrawCall time-consuming data of the GPU rendering pipeline into a visual heat map to quickly locate high-time-consuming DrawCalls and their associated models / materials.

[0098] Operation process:

[0099] Data input: Receive the original rendering frame mat_frame (RGBA format) and the DrawCall time-consuming data collected by the performance probe Pass (structure: TMap<int32, float>, key: DrawCall index, value: time-consuming (ms)).

[0100] Data normalization:

[0101] Calculate the minimum value (t_min) and maximum value (t_max) of all DrawCall time-consuming, and calculate the normalization coefficient:

[0102] scale = 1.0 / (t_max - t_min) (if t_max == t_min, set it to 1.0).

[0103] For each DrawCall's time cost t, calculate the normalized value: normalized_t = (t - t_min) * scale (range: 0~1).

[0104] Color mapping:

[0105] Map the normalized value to RGB color using Jet color map:

[0106] normalized_t ∈ [0, 0.33]: blue (low time cost) → (0, 0, 255 * (1 - 3*normalized_t));

[0107] normalized_t ∈ [0.33, 0.66]: cyan → purple transition → (0, 255*(3*normalized_t- 1), 255*(1.5 - 3*|normalized_t - 0.5|));

[0108] normalized_t ∈ [0.66, 1]: yellow → red (high time cost) → (255*(3*normalized_t -2), 255*(2 - 3*normalized_t), 0).

[0109] Heat map overlay:

[0110] Fill the color corresponding to the DrawCall time cost into the Alpha channel or a separate channel of the original frame according to the pixel area covered by the DrawCall (recorded by the geometry marking pass, the mapping relationship between model ID / material ID and pixel coordinates), to generate a heat map overlay layer.

[0111] # Rendering quality detection module (i.e. analyzing image data)

[0112] exposure_defects = detect_exposure_issue(mat_frame)

[0113] Function 1: Detect abnormal pixel value areas in the image caused by excessive or insufficient light (overexposure: pixel value close to 255; underexposure: close to 0).

[0114] Operation process:

[0115] Channel separation: separate the input frame mat_frame into R, G, B single channels (cv::split).

[0116] Histogram statistics:

[0117] For each channel, calculate the 256-level grayscale histogram (cv::calcHist) to count the number of pixels at each intensity level (0~255).

[0118] Calculate the high-light pixel ratio (number of pixels with intensity ≥224 / total number of pixels) and the dark-pixel ratio (number of pixels with intensity ≤32 / total number of pixels) for each channel.

[0119] Threshold determination:

[0120] Overexposure trigger condition: high-light pixel ratio >15% in any channel (empirical threshold, can be adjusted through training data).

[0121] Underexposure trigger condition: dark-pixel ratio >15% in any channel.

[0122] Region marking:

[0123] For overexposed regions: binarize the high-light pixels in R / G / B channels (cv::threshold, threshold=224) to extract the coordinate set.

[0124] For underexposed regions: similarly, extract the coordinate set of pixels with intensity ≤32.

[0125] Result output: return the mask of overexposed / underexposed regions (cv::Mat format, abnormal area is 255, others are 0) and the ratio data.

[0126] Function two, detect the dramatic changes between consecutive frames at the same position (screen flicker, moire).

[0127] Frame difference method: calculate the absolute difference between adjacent frames and extract the area with large intensity changes.

[0128] Optical flow method: track the intensity changes of moving areas.

[0129] Function three, picture tearing detection, detect the picture misalignment caused by vertical synchronization failure.

[0130] Top-bottom half frame comparison: divide each frame into top and bottom parts and calculate the difference between the two parts.

[0131] Edge matching: use Canny edge detection to detect the continuity of the top and bottom edges.

[0132] transparency_artifact = detect_transparency_artifact(mat_frame)

[0133] Function four: detect the edge black border, color fault, etc. caused by the rendering sequence error of semi-transparent objects.

[0134] Operation flow:

[0135] Translucent region extraction:

[0136] Based on the Alpha channel threshold (e.g. Alpha < 0.9), extract the translucent region (cv::Mat format, transparent region is 255) by cv::inRange.

[0137] Correct rendering result generation (OIT pre-rendering):

[0138] Re-render the translucent object using the linked list OIT algorithm (Order Independent Transparency):

[0139] Maintain a depth linked list for each translucent fragment in the GPU, sorted by depth from back to front.

[0140] Mix the color in the sorted order (FinalColor = AccumulatedColor + CurrentColor *CurrentAlpha).

[0141] Align the OIT pre-rendering result (cv::Mat format) with the original rendering frame (map the coordinates by the model ID of the geometry pass).

[0142] Difference comparison:

[0143] Calculate the mean square error (MSE) between the original frame and the OIT pre-rendering result:

[0144] Set the threshold (e.g. MSE>50), mark the region with MSE exceeding the threshold as a translucent defect.

[0145] Result output: return the mask of the defect region and the MSE value. The specific implementation code is as follows:

[0146] return {"performance": drawcall_heatmap,

[0147] "quality": {

[0148] "exposure": exposure_defects,

[0149] "transparency": transparency_artifact}}

[0150] Function five, rendering artifact detection (jaggies, moire)

[0151] Edge detection + frequency analysis: use Sobel / Canny to detect edges, analyze the sharpness of the edges (jagged edges appear as short and broken edges).

[0152] Frequency domain transform: transform the image to the frequency domain by FFT, the concentrated peak value of the high frequency region may correspond to the artifact.

[0153] This part also includes performance anomaly analysis module.

[0154] Insert GPU time query instructions between each DrawCall to obtain the execution time of each DrawCall, and transform the data into color and write it into the corresponding pixel, and accumulate in each DrawCall, finally by observing the generated image can determine the pixel that consumes too much, and can quickly obtain the material or model that may cause performance anomaly through the corresponding relationship between the pre-generated pixel and the model and the material.

[0155] Function six, real-time detection and analysis data

[0156] Integrate the entire module into the rendering process of UE5, control the enable and disable of the module through instructions, so as to realize real-time data analysis during program running. Without recording data during running and offline data analysis as in traditional scheme.

[0157] Timely execute the above process for analysis, collect analysis results, generate visual diagnostic report, and trigger exception repair work order system.

[0158] Through the above embodiments and the accompanying Figure 2 It can be seen that the scheme also has the following effects:

[0159] Can skip the originally complex manual debugging steps (exception reproduction-grab single-frame rendering data-analyze rendering data), and automatically analyze the entire rendering process.

[0160] Cross-frame exception analysis capability, the original analysis process can only analyze single-frame data, if the exception is not caused by the current frame, it is difficult to locate, the scheme can locate the exception source caused by non-current frame through continuous multi-frame data analysis, such as material flicker or ghosting.

[0161] High positioning accuracy, through the method of marking the model and the material by pixels, the positioning accuracy of the exception reason is improved to a single material or model, reducing the troubleshooting time.

[0162] In the above examples, the programming code parts used, the format and the symbols used are the usual writing of programming codes, mainly play the role of identification or interpretation, etc. can be replaced by other ways, those skilled in the art can understand the meaning represented by them without objection, and there will be no unclear problems.

[0163] Those skilled in the art should understand that those skilled in the art can realize variations in combination with the prior art and the above examples, and such variations do not affect the essential content of the scheme, which will not be described here.

[0164] It should be understood that the present scheme is not limited to the above specific embodiments, and the devices and structures not fully described should be understood as implemented in the ordinary way in the art; any person skilled in the art can make many possible changes and modifications to the above disclosed methods and technical contents without departing from the scope of the present technical scheme, or modify equivalent examples of equivalent changes, which does not affect the essential content of the present scheme. Therefore, any simple modification, equivalent change and modification made to the above examples according to the technical essence of the present scheme without departing from the content of the present technical scheme still belongs to the protection scope of the present technical scheme.< / frendertarget>

Claims

1. An automated diagnostic system for rendering anomalies based on UE5 pixel readback and OpenCV, characterized by: It includes a pipeline extension layer, a data capture layer, and an intelligent analysis layer; wherein, the pipeline extension layer constructs a rendering pipeline extension model and collects RenderTarget data; the rendering pipeline extension model includes embedding three types of custom DebugPasses into the UE5 standard rendering pipeline, namely: The geometric marker Pass is used to record the RGBA encoding information of the model instance ID / material ID; Performance probes (Pass) are used to insert GPU timestamp markers during critical rendering phases; The feature enhancement pass is used to enhance the visual features of potentially anomalous regions; the specific process of collecting RenderTarget data is as follows: First, add a custom DebugPass after a specific rendering Pass according to different requirements; second, collect the RenderTargets of the custom DebugPass, the built-in UE5 DebugPass, and the normal rendering Pass. The data capture layer uses a double buffering mechanism to achieve asynchronous reading of RenderTarget; the intelligent analysis layer constructs a multi-dimensional anomaly detection model based on OpenCV and analyzes image data to obtain anomaly analysis results.

2. The automated diagnostic system for rendering anomalies based on UE5 pixel readback and OpenCV as described in claim 1, characterized in that: The data capture layer reads back the data collected in the RenderTarget and saves it to the UTexture2D object; and uses the data in the UTexture2D to create the corresponding cv::mat object.

3. The automated diagnostic system for rendering anomalies based on UE5 pixel readback and OpenCV as described in claim 2, characterized in that: The process of analyzing the image data includes: Scene color anomaly detection before post-processing, drastic changes in the same position between consecutive frames, screen tearing detection, semi-transparent rendering sorting anomaly detection, and rendering artifact detection.

4. The automated rendering anomaly diagnosis system based on UE5 pixel readback and OpenCV as described in claim 3, characterized in that: Drastic changes at the same location between consecutive frames include screen flicker and moiré patterns, which are detected using a combination of inter-frame difference and optical flow methods.

5. The automated rendering anomaly diagnosis system based on UE5 pixel readback and OpenCV according to claim 3, characterized in that: Rendering artifact detection includes jagged edges and moiré patterns. Edge detection and frequency analysis methods are used to detect the sharpness of edges, and artifacts are detected through frequency domain transformation.

6. The automated rendering anomaly diagnosis system based on UE5 pixel readback and OpenCV according to claim 1, characterized in that: It also includes a performance anomaly analysis module, and the specific analysis process is as follows: A GPU time query instruction is inserted between each DrawCall to obtain the execution time of each DrawCall, and the data is transformed into color and written into the corresponding pixel. This data is accumulated in each DrawCall, and finally, pixels that consume too much power are identified by observing the generated image. Furthermore, materials or models that may cause performance abnormalities are identified by using the pre-generated correspondence between pixels, models, and materials.

Citation Information

Patent Citations

  • Detection method and system for digital human rendering video

    CN115471489A

  • Video GPU (Graphic Processing Unit) decoding image anomaly detection method and system based on shader

    CN116647672A