A state prediction based method, system, device and medium for suppressing visual judder

Abstract

The application relates to a state prediction-based visual scene sawtooth flicker suppression method, system, device and medium, which comprises the following steps: acquiring a picture rendered by a visual scene engine at a current moment as a current original frame and a picture rendered by the visual scene engine at a previous moment as a previous output frame; estimating a transformation relationship between the current original frame and the previous output frame to obtain a transformation matrix; predicting the previous output frame by using the transformation matrix to generate a predicted frame; comparing the current original frame with the predicted frame to detect a flicker area and generate a flicker map; generating a compensation map based on the flicker map; fusing the current original frame and the predicted frame by using the compensation map to generate a final output frame; and taking the final output frame as the previous output frame, taking a picture rendered by the visual scene engine at a next moment as the current original frame, and repeatedly executing the processing. The application can effectively suppress visual scene flicker, maintain picture definition, dynamically detect flicker, has high calculation efficiency and does not require additional model resources.

Description

Technical Field

[0001] This invention belongs to the field of visual image processing technology, specifically relating to a method, system, device, and medium for suppressing visual jagged flicker based on state prediction. Background Technology

[0002] Flight simulators are highly realistic flight training devices, and training time on flight simulators can be partially counted towards a pilot's required flight experience time. The visual system is a key component of a flight simulator, used to simulate the scenery seen during flight. Within the visual system, high-frequency flickering is likely to occur near certain objects with high contrast, such as the edges of buildings.

[0003] The root cause of this flickering is that during image output, high-resolution images must be rasterized into discrete pixels. For tiny objects or lines in the image, if their image contains an area smaller than one pixel, rasterization will inevitably introduce a rasterization error in that area, meaning these pixels will either be fully displayed or completely hidden. These pixels appear and disappear as the viewpoint moves, creating the flickering effect. When the image contains many sharp lines (such as clusters of buildings), the lines gradually become thinner and smaller as their distance from the viewpoint increases, causing this phenomenon to recur. During rasterization, the generation and disappearance of pixels... Figure 1 As shown; the magnified comparison of two consecutive frames in the actual visual engine is as follows: Figure 2 As shown, in the same area, there is a discrepancy in the brightness and darkness of pixels, and this phenomenon is particularly noticeable when there are many detailed buildings in the vicinity of the area.

[0004] There are currently two main types of solutions to the flickering problem in visual systems:

[0005] 1. Overall Image Smoothing: This involves blurring the entire image or increasing the intensity of anti-aliasing algorithms to smooth the transition between fine lines and surrounding pixels. Methods include Multi-Sample Anti-Aliasing (MSAA), Fast Approximate Anti-Aliasing (FXAA), and Temporal Anti-Aliasing (TXAA). However, these methods have drawbacks. They cannot distinguish between foreground and background, nor can they identify flickering areas. Smoothing is based solely on the current frame, which can easily lead to a decrease in overall image sharpness. Furthermore, simple smoothing often has limited effectiveness against strong flickering. In scenarios requiring high-precision recognition, such as flight simulations, the decrease in overall image sharpness can negatively impact training results.

[0006] 2. Multi-level Detail Representation: This method employs MIPMAPs (Multi-level Texture Mapping) or LOD (Levels of Detail). The basic principle is to pre-create multiple models or textures of varying precision and switch between them based on the viewpoint distance. When the viewpoint is close, high-precision models and textures are used. Because the distance is short, lines on the model and texture require multiple pixels to be described, thus eliminating noticeable flickering. As the viewpoint moves further away, lower-precision models and textures are loaded. These lower-precision models and textures have less detail, only outlines, and also require multiple pixels to be drawn, thus avoiding flickering. However, this approach suffers from high labor costs and low efficiency. Firstly, the effectiveness of the method depends on the number of model precision levels. The more refined the level division, the better the effect. However, it requires the pre-production of multiple sets of textures and models. It consumes a lot of resources, has high requirements for computer performance, and increases the complexity and maintenance cost of the system. Furthermore, the switching between different precision levels may produce visible "jumps", affecting visual continuity. It also requires a lot of manual work to produce textures and models of various precision levels, which is labor-intensive and difficult to automate.

[0007] Therefore, existing methods all require modifications to the current visual display framework, such as inserting a smoothing filter stage or replacing the model. This requires significant changes to the existing project, making it difficult to deploy and roll back quickly, which is not conducive to system maintenance and upgrades. Furthermore, it cannot accurately locate and handle flickering areas, resulting in problems such as over-processing leading to overall blurring or under-processing causing flickering to persist. Summary of the Invention

[0008] To overcome the problems existing in the prior art, the present invention provides a method, system, device and medium for suppressing visual sawtooth flicker based on state prediction, which is used to overcome the existing defects.

[0009] A method for suppressing visual sawtooth flicker based on state prediction, the method comprising the following steps:

[0010] S1. Obtain the current frame rendered by the view engine as the current raw frame, and the previous frame rendered by the view engine as the previous output frame.

[0011] S2. Estimate the transformation relationship between the current original frame and the previous output frame to obtain the transformation matrix;

[0012] S3. Use the transformation matrix to predict the previous output frame and generate a prediction frame;

[0013] S4. Compare the current original frame with the predicted frame, detect the flickering area, and generate a flickering map;

[0014] S5. Generate a compensation map based on the flicker map;

[0015] S6. Use a compensation map to fuse the current original frame and the predicted frame to generate the final output frame;

[0016] S7. Using the final output frame as the previous output frame, and the image rendered by the visual engine at the next moment as the current original frame, repeat steps S2-S6.

[0017] In addition to the aspects and any possible implementations described above, a further implementation is provided in which S1 further includes: processing the current original frame image and the previous output frame image to obtain images of the same size and both converted to the RGB color space, thereby obtaining the converted current original frame image and the previous output frame image.

[0018] In addition to the aspects and any possible implementations described above, a further implementation is provided, wherein S2 specifically includes:

[0019] S21. Use the ORB algorithm to calculate the key points and descriptors of the current original frame and the previous output frame;

[0020] S22. Use the BFMatcher algorithm to match feature points in the current original frame and the previous output frame;

[0021] S23. Based on the similarity between the descriptors of the current original frame and the descriptors of the previous output frame, find the feature point pairs corresponding to the current original frame and the previous output frame.

[0022] S24. Sort by matching distance and select a certain number of best matching pairs;

[0023] S25. Perform matching filtering on the best matching pairs to obtain matching feature point pairs;

[0024] S26. Calculate the transformation matrix based on the matched feature point pairs.

[0025] As described above, and in accordance with any possible implementation, an implementation is further provided, wherein S3 specifically includes:

[0026] S31. For each pixel coordinate in the previous output frame, apply the transformation matrix to calculate the new coordinates of the corresponding pixel in the prediction frame;

[0027] S32. The inverse matrix of the previous output frame and the transformation matrix is ​​calculated using the interpolation method to obtain the pixel value at the new coordinates of the predicted pixel.

[0028] As described above, and in accordance with any possible implementation, an implementation is further provided, wherein S4 includes:

[0029] S41. Perform grayscale conversion on the current original frame image and the predicted frame image respectively to obtain the grayscale image of the current original frame and the grayscale image of the predicted frame;

[0030] S42. Calculate the absolute difference between the current original frame grayscale image and the predicted frame grayscale image to obtain the grayscale difference image;

[0031] S43. If the absolute difference is greater than the set threshold, it is considered that there is a flickering area in the current original frame. The grayscale difference map is subjected to noise suppression processing to obtain the processed flickering map.

[0032] In addition to the aspects and any possible implementations described above, an implementation is further provided, wherein S5 includes: S51. Removing regions below a preset minimum flicker threshold from the flicker map to obtain a preliminary compensation map;

[0033] S52. Normalize and filter the preliminary compensation image to obtain a compensation image in which all pixel values ​​are less than or equal to 1.

[0034] In addition to the aspects and any possible implementations described above, a further implementation is provided in which the RGB three-channel compensation map obtained by expanding the compensation map in S6 is used to fuse the current original frame and the predicted frame.

[0035] The present invention also provides a visual sawtooth flicker suppression system based on state prediction, the system being used to implement the method, comprising the following modules:

[0036] The acquisition module is used to acquire the image rendered by the visual engine at the current moment as the current raw frame, and the image rendered by the visual engine at the previous moment as the previous output frame.

[0037] The estimation module is used to estimate the transformation relationship between the current original frame and the previous output frame to obtain the transformation matrix;

[0038] The prediction module is used to predict the previous output frame using a transformation matrix and generate a prediction frame.

[0039] The comparison module is used to compare the current original frame with the predicted frame, detect flickering areas, and generate a flickering map;

[0040] The first generation module is used to generate a compensation map based on the flicker map;

[0041] The second generation module is used to fuse the current original frame and the predicted frame using a compensation map to generate the final output frame.

[0042] The continuing processing module is used to repeatedly perform processing, taking the final output frame as the previous output frame and the scene rendered by the visual engine at the next moment as the current original frame.

[0043] The present invention also provides an electronic device, the electronic device comprising:

[0044] Memory, which stores executable instructions;

[0045] A processor that executes the executable instructions in the memory to implement the method.

[0046] The present invention also provides a computer storage medium storing a computer program, the computer program being executed by a processor to implement the method described.

[0047] Beneficial effects of the present invention

[0048] The present invention provides a state prediction-based method for suppressing visual jagged flicker, comprising the following steps: acquiring the current scene rendered by the visual engine as the current original frame, and the scene rendered by the visual engine at the previous moment as the previous output frame; estimating the transformation relationship between the current original frame and the previous output frame to obtain a transformation matrix; using the transformation matrix to predict the previous output frame to generate a prediction frame; comparing the current original frame and the prediction frame to detect flickering areas and generate a flicker map; generating a compensation map based on the flicker map; fusing the current original frame and the prediction frame using the compensation map to generate a final output frame; and repeating the process using the final output frame as the previous output frame and the scene rendered by the visual engine at the next moment as the current original frame.

[0049] The method of the present invention has the following beneficial effects:

[0050] 1. Effectively suppresses visual flicker: It can accurately identify and locate flickering areas and process them in a targeted manner; it has a particularly good effect on suppressing flickering in complex scenes such as building clusters; it smooths temporal changes through visual persistence and reduces visual discomfort caused by flickering;

[0051] 2. Maintain image clarity: Suppresses flicker without affecting the display quality of non-flickering areas; unlike global blurring, it does not reduce overall image clarity; precise area recognition ensures that compensation is only applied to the areas that need it.

[0052] 3. Flexible and convenient deployment: It serves as an additional component at the very end of the visual system output; it does not require modification of the original rendering architecture and process; it can be turned on or off at any time, facilitating rapid deployment and rollback; and it can be seamlessly integrated into existing visual systems.

[0053] 4. Dynamic flicker detection: It can automatically detect the location and intensity of flickering; no need to pre-mark or set the flickering area; it adaptively adjusts the processing area as the scene changes;

[0054] 5. High computational efficiency: It uses the efficient ORB feature detection algorithm, which has lower computational complexity than algorithms such as SIFT and SURF; the processing is optimized and suitable for real-time applications; compared with multi-level detail model methods, it has lower storage and computational overhead.

[0055] 6. No additional model resources required: It eliminates the need to pre-create models or textures of varying precision; reduces the cost of producing additional art resources; and lowers the complexity and maintenance costs of the visual system. This invention effectively solves the jagged edge flickering problem while maintaining the performance and image quality of the visual system, thereby improving the simulation effect and user experience of the flight simulation system. Attached Figure Description

[0056] Figure 1 is a schematic diagram of the appearance and disappearance of pixels during the rasterization process in the prior art;

[0057] Figure 2 is a schematic diagram showing a partial enlarged view of the buildings surrounding the airport using existing technology;

[0058] Figure 3 is a schematic diagram of the current original frame and the predicted frame of the present invention;

[0059] Figure 4 is a schematic diagram of the scintillation pattern of the present invention;

[0060] Figure 5 This is a schematic diagram of the compensation diagram of the present invention;

[0061] Figure 6 This is a schematic diagram of flicker suppression according to the present invention;

[0062] Figure 7 This is a flowchart of the method of the present invention. Detailed Implementation

[0063] To better understand the technical solution of this invention, the content of this invention includes, but is not limited to, the specific embodiments described below. Similar technologies and methods should be considered within the scope of protection of this invention. To make the technical problems to be solved, the technical solutions, and advantages of this invention clearer, a detailed description will be provided below in conjunction with the accompanying drawings and specific embodiments.

[0064] It should be understood that the embodiments described in this invention are merely some embodiments of this invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are within the scope of protection of this invention.

[0065] like Figure 7 As shown, the present invention provides a state prediction-based method for suppressing visual sawtooth flicker, the method comprising the following steps:

[0066] S1. Obtain the current frame rendered by the view engine as the current raw frame, and the previous frame rendered by the view engine as the previous output frame.

[0067] S2. Estimate the transformation relationship between the current original frame and the previous output frame to obtain the transformation matrix;

[0068] S3. Use the transformation matrix to predict the previous output frame and generate a prediction frame;

[0069] S4. Compare the current original frame with the predicted frame, detect the flickering area, and generate a flickering map;

[0070] S5. Generate a compensation map based on the flicker map;

[0071] S6. Use a compensation map to fuse the current original frame and the predicted frame to generate the final output frame;

[0072] S7. Using the final output frame as the previous output frame, and the image rendered by the visual engine at the next moment as the current original frame, repeat steps S2-S6.

[0073] Furthermore, S1 also includes: processing the current original frame image and the previous output frame image to obtain images of the same size and both converted to the RGB color space, thereby obtaining the converted current original frame image and the previous output frame image.

[0074] Furthermore, S2 specifically includes:

[0075] S21. Use the ORB algorithm to calculate the key points and descriptors of the current original frame and the previous output frame;

[0076] S22. Use the BFMatcher algorithm to match feature points in the current original frame and the previous output frame;

[0077] S23. Based on the similarity between the descriptors of the current original frame and the descriptors of the previous output frame, find the feature point pairs corresponding to the current original frame and the previous output frame.

[0078] S24. Sort by matching distance and select a certain number of best matching pairs;

[0079] S25. Perform matching filtering on the best matching pairs to obtain matching feature point pairs;

[0080] S26. Calculate the transformation matrix based on the matched feature point pairs.

[0081] Furthermore, S3 specifically includes:

[0082] S31. For each pixel coordinate in the previous output frame, apply the transformation matrix to calculate the new coordinates of the corresponding pixel in the prediction frame;

[0083] S32. The inverse matrix of the previous output frame and the transformation matrix is ​​calculated using the interpolation method to obtain the pixel value at the new coordinates of the predicted pixel.

[0084] Furthermore, S4 includes:

[0085] S41. Perform grayscale conversion on the current original frame image and the predicted frame image respectively to obtain the grayscale image of the current original frame and the grayscale image of the predicted frame;

[0086] S42. Calculate the absolute difference between the current original frame grayscale image and the predicted frame grayscale image to obtain the grayscale difference image;

[0087] S43. If the absolute difference is greater than the set threshold, it is considered that there is a flickering area in the current original frame. The grayscale difference map is subjected to noise suppression processing to obtain the processed flickering map.

[0088] Further, S5 includes: S51. Removing regions below a preset minimum flicker threshold from the flicker map to obtain a preliminary compensation map;

[0089] S52. Normalize and filter the preliminary compensation image to obtain a compensation image in which all pixel values ​​are less than or equal to 1.

[0090] Furthermore, in S6, the RGB three-channel compensation map obtained by expanding the compensation map is used to fuse the current original frame and the predicted frame.

[0091] Specifically, the implementation process of this invention is as follows:

[0092] This invention effectively suppresses jagged flicker in visual systems by intelligently identifying and precisely processing flickering areas without affecting overall image clarity. The method includes the following steps:

[0093] 1. Image Acquisition and Preparation

[0094] First, obtain two key data points from the visual engine:

[0095] The current raw frame (frame_src): The output image of the raw scene rendered by the view engine at the current moment, also known as the current frame;

[0096] The previous output frame, frame_opt_last, is the original image output by the view engine at the previous moment.

[0097] These two data sets form the basis for subsequent processing. The current original frame provides the latest visual information, while the previous output frame serves as the basis for prediction. This method first processes both images to the same size and converts them to the RGB color space to ensure consistency in subsequent processing.

[0098] 2. Image Transformation Parameter Estimation

[0099] The transformation relationship between the current original frame and the previous output frame is analyzed using a feature point matching algorithm, and a transformation matrix H is calculated. This transformation matrix describes the viewpoint movement from the previous frame to the current frame. The specific steps are as follows:

[0100] 2.1 Feature Point Detection and Description: The ORB (Oriented Fast and Rotated BRIEF) algorithm is used to detect feature points and descriptors in two frames of images. The ORB algorithm is a fast feature detection algorithm that can effectively identify corner points and features in images. The specific calculation and solution process adopts existing common methods, which will not be elaborated in this invention. For each frame of image, i.e., the current original frame and the previous output frame, the ORB algorithm is used to calculate and output the keypoints and descriptors of the current original frame and the previous output frame, respectively. Keypoints represent the feature positions in the image, and descriptors are feature vector descriptions of the region surrounding each keypoint.

[0101] This invention uses the ORB algorithm to obtain the feature points and descriptors detected in the previous output frame. and feature points and descriptors detected in the original frame .

[0102] Alternatively, in addition to using the ORB algorithm to detect feature points, this invention can also employ SIFT (Scale-Invariant Feature Transform), which offers better scale and rotation invariance and can handle more complex scene transformations. Alternatively, the ORB algorithm can be replaced by SURF (Speeded-Up Robust Features), an accelerated version of SIFT with performance between SIFT and ORB, balancing accuracy and speed for scenarios requiring high accuracy but still needing a certain level of real-time performance.

[0103] The implementation architecture of this invention has reserved an interface for the SIFT method, which can be switched according to actual needs.

[0104] 2.2 Feature point matching: The BFMatcher (Brute-Force Matcher) algorithm is used to match feature points in two frames of images; based on the similarity of descriptors, corresponding feature point pairs in the two frames of images are found; the best matching point pairs are selected according to the matching distance (the top 10% of the best matches are selected); unreliable feature points are filtered out during the matching process, and only high-quality matching pairs are retained.

[0105] This invention is based on the existing Brute-Force Matcher method, which uses this method to obtain a preliminary set of matched feature point pairs. For the set of feature points Matching results sorted by distance And the matching results are filtered to obtain high-quality matching feature point pairs. The screening process is performed by selecting the top 10% of items with the smallest distance.

[0106] 2.3 Calculation of the transformation matrix:

[0107] Based on the matched feature point pairs, the transformation matrix H is calculated. This invention employs two transformation types: affine transformation and perspective transformation. Perspective transformation can handle more complex viewpoint changes, including rotation, translation, scaling, and perspective effects. Affine transformation is used for simple translation, rotation, and scaling scenes, with lower computational cost.

[0108] Depend on and The set of feature point coordinates in the previous frame is obtained by using perspective transformation. ;Depend on and Perspective transformation is used to obtain the set of coordinates of the corresponding feature points in the current frame. ;Depend on and The transformation matrix H is obtained by using perspective transformation or affine transformation. This transformation matrix H is a 3×3 matrix used to describe the spatial transformation relationship from the previous frame to the current original frame, and its expression is as follows:

[0109] (1),

[0110] in: Used to control rotation and scaling; Used to control translation; Used to control perspective effects.

[0111] Key parameters in the feature detection and matching process include: Number of feature points: This affects the accuracy and speed of transform estimation. This invention uses 1000 feature points, which can be adjusted according to processing speed and accuracy requirements; Feature matching quality threshold: This invention selects the top 10% of the best matching points to reduce the impact of outliers while ensuring matching quality; Transformation matrix calculation method: The default is to use the RANSAC (Random Sample Consensus) algorithm. This invention uses 100 iterations to effectively eliminate incorrect matches; RANSAC reprojection threshold: This invention uses 5.0 pixels to determine whether a feature point is an interior point. By adjusting these parameters, optimal flicker suppression can be achieved in different scenarios, balancing the relationship between flicker suppression and image sharpness.

[0112] Preferably, in addition to perspective transformation, the present invention can also use local transformation, which can segment the image into multiple local regions and estimate the transformation matrix for each region separately; it is suitable for situations where the transformation differences of objects at different depths are large in complex scenes, and can handle scenes with strong perspective effects more accurately.

[0113] This invention implements both affine transformation and perspective transformation, which can be selected through configuration.

[0114] 3. Generate prediction frames

[0115] This step generates the predicted image for the current frame based on the previous output frame and the transformation matrix H. The calculation process is as follows:

[0116] 3.1 Perspective Transformation Application: For each pixel's coordinates or position (x, y) in the previous output frame, the transformation matrix H is applied to calculate its new coordinates or position (X', Y') in the prediction frame. The pixel value of the prediction frame at this new position is obtained through interpolation, thus obtaining or generating the prediction frame. The expression for the perspective transformation used to calculate the new coordinates or position is:

[0117] (2)

[0118] (3)

[0119] in: These are the pixel coordinates from the previous output frame; It predicts the corresponding pixel coordinates in the frame. It is the scaling factor for perspective transformation, used to convert three-dimensional homogeneous coordinates back to two-dimensional coordinates. It is a known quantity and is set as needed.

[0120] Therefore, the coordinates of all pixels in the predicted frame can be calculated using the above formula. The predicted frame is an image inferred from the previous output frame based on the motion trajectory using a transformation matrix H. It is very close to the current original frame image, but will produce pixel-level differences in flickering areas. Figure 3 As shown, the left side is the current original frame image, and the right side is the predicted frame image. The current original frame and the predicted frame have slight differences in local pixels. The predicted frame is not directly from the rendering output of the image engine (the software that generates the image), but is generated by moving the image of the previous frame.

[0121] 4. Scintillation Detection and Scintillation Pattern Generation

[0122] This invention detects flickering regions and generates a flicker map by comparing the current original frame and the predicted frame. The specific steps are as follows:

[0123] 4.1 Grayscale Conversion:

[0124] The original frame and the predicted frame are converted into grayscale images to simplify the calculation and focus on brightness changes. The conversion formula is as follows:

[0125] (4)

[0126] (5)

[0127] in: Represents the grayscale image of the current original frame; Represents the grayscale image of the predicted frame; This represents a function that converts a color image to a grayscale image using existing transformation methods.

[0128] 4.2 Difference Calculation: The absolute difference between two grayscale images is calculated to obtain an initial flicker map or grayscale difference map. A larger absolute difference indicates a greater deviation between the prediction and the actual value, suggesting a higher probability that the region in the current original frame is a flickering region. Therefore, this invention uses this method to detect the presence of flickering regions in an image. Existing technologies, such as the Kalman filter-based anti-aliasing method in Technology 1 (CN119583811A) and the temporal pixel tracking anti-aliasing and post-processing rendering method in Technology 2 (CN118736093A), directly address flickering without first detecting or judging its existence. This invention overcomes the shortcomings of the existing technologies by first determining whether flickering exists, proceeding with the next process only if it exists, and performing no processing if it does not. This method effectively targets flickering, especially effectively suppressing flickering in complex scenes such as building clusters, thereby accelerating processing speed and improving computational efficiency.

[0129] The formula for calculating the grayscale difference image is as follows: (6)

[0130] in, Represents the grayscale image of the current original frame. Grayscale image of the predicted frame Grayscale difference image between; This indicates an absolute value operation, which calculates the difference between the grayscale image of the current original frame and the grayscale image of the predicted frame.

[0131] Alternatively, the present invention can also employ time series analysis for flicker detection. This analysis method not only compares the current frame with the predicted frame, but also analyzes historical data from multiple frames. By statistically analyzing the time change pattern of each pixel, it can more accurately identify flickering areas and distinguish between one-time changes and periodic flickering. Or, frequency domain analysis can be used: the time domain is converted to the frequency domain for each region, and the spectral characteristics are analyzed to identify flickering. This can more accurately distinguish between flickering and normal conditions. All of these methods can accurately detect flickering.

[0132] 4.3 Noise Suppression: Gaussian blur is applied to the grayscale difference image to reduce the impact of noise. This invention uses Gaussian blur to smooth the image and reduce the interference of isolated noise points on flicker detection. The expression used for Gaussian blur processing is as follows:

[0133] (7)

[0134] in, This represents the processed flickering image; Indicates the Gaussian blur operation; This represents the size of the Gaussian kernel, which is set to (5, 5) here; The standard deviation of the Gaussian distribution is represented by 1.5 in this invention.

[0135] 4.4 Value Range Limitation:

[0136] Set blinking pattern The pixel value range is [0, ...]. Between, among which, This is the preset upper limit of flash intensity, with a default value of 15, calculated by the following formula:

[0137] (8)

[0138] in The function is used to limit the maximum threshold, or upper limit, of the flicker intensity of pixel values ​​in a flicker map. ;Should The saturation level for flicker detection is controlled; a higher value results in greater discrimination. This invention sets it to 15, but it can be adjusted according to the actual flicker situation. Smaller values ​​make the differences in flicker intensity less noticeable, resulting in smaller differences in compensation strength; while larger values ​​increase the discrimination of flicker intensity, making the compensation more targeted.

[0139] The value represents the flicker intensity at the corresponding position in the flicker image: the larger the value, the stronger the flicker at that position. In this way, the present invention can accurately locate flickering areas in an image, resulting in a flicker image as shown below. Figure 4 As shown, black areas indicate intense flickering, gray areas indicate less flickering, and white areas indicate no flickering.

[0140] 5. Generate compensation map

[0141] Based on the flicker map obtained in step 4, a compensation map is generated in this step to control the fusion ratio between the current original frame and the predicted frame. The compensation map is calculated as follows:

[0142] 5.1 Flicker Threshold Processing: Subtract the preset minimum flicker intensity threshold from the flicker intensity at the corresponding pixel position in the flicker map. ,Should The preset minimum flicker intensity threshold is set to 10 in this invention; areas in the flicker map below this threshold are considered non-flicker areas requiring no compensation, and their values ​​are set to 0, meaning these areas are considered to be free of flicker. The preliminary compensation map is generated as follows:

[0143] (9)

[0144] in: A preliminary compensation diagram; The function ensures that the values ​​in the compensation graph are not less than 0.

[0145] Among them, the minimum threshold of flicker intensity As the sensitivity of flicker detection is controlled, the smaller the value, the more sensitive the detection. This invention sets it to 10, which is adjusted according to the actual flicker situation. Setting a smaller value will cause more areas to be identified as flickering areas, resulting in over-smoothing. On the other hand, setting a larger value will reduce the identified flickering areas, which may lead to insufficient flicker suppression.

[0146] 5.2 Normalization process:

[0147] The preliminary compensation map is processed in one step to normalize the result to the range [0,1].

[0148] Normalization ensures the compensation map has a uniform scale, facilitating subsequent processing. (Normalized compensation map) The formula for obtaining is as follows:

[0149] (10)

[0150] Where: denominator Indicates the valid range of the blink value.

[0151] 5.3 Calculation of compensation coefficient:

[0152] Final compensation diagram The calculation is expressed by the following formula:

[0153] (11)

[0154] in: This is the maximum filtering coefficient, which controls the maximum contribution of the predicted frame. The smaller the value, the stronger the suppression effect. In this invention, it is set to 0.1, meaning that in the area of ​​strongest flicker, the current original frame retains only 10% of the information, while the predicted frame contributes 90% of the information. Smaller values ​​will make flicker suppression stronger, but may cause motion blur; larger values ​​will weaken the flicker suppression effect, but can retain more original details.

[0155] The final compensation diagram ensures that its value range is within Between; that is, the pixel values ​​of the final compensated image range from The final compensated image is represented using a decimal grayscale image, where 0 represents white and 1 represents black. The value range is the range of its pixel values.

[0156] Pixel values ​​are close The location of a pixel in the current original frame indicates a region of strong flickering, requiring significant compensation using a predicted frame; locations with pixel values ​​close to 1 indicate flicker-free regions, where the content of the current original frame is preserved; and locations where the pixel value is close to 1 indicate flicker-free regions. Values ​​between 1 and 0 indicate areas of slight flickering that require partial compensation.

[0157] In this way, the compensation intensity can be dynamically adjusted according to the degree of flicker, achieving precise flicker suppression. An example of a compensation map calculation... Figure 5 As shown in the figure, darker colors indicate that more predicted frames are used in subsequent fusion, rather than the actual current original frames. Pure black areas represent the use of predicted frames entirely, while pure white areas represent the use of actual current original frames entirely.

[0158] 6. Image fusion output

[0159] Finally, this invention fuses the current original frame and the predicted frame based on the final compensation map to generate...

[0160] frame_output. The fusion formula is as follows:

[0161] (12)

[0162] in, This is the current original frame; It is a predicted frame; This is a three-channel extended compensation map obtained by expanding the final compensation map. The same compensation value is applied to each RGB channel. The extended final compensation map is expressed as follows:

[0163] (13)

[0164] in, This indicates the operation of copying a single-channel image to three channels; 3 represents the target number of channels (corresponding to the three RGB channels), and the operation is performed once on each of the three RGB color channels, for a total of three times.

[0165] The present invention uses a final compensation map to fuse the current original frame and the predicted frame. Specifically, in the flickering area of ​​the current original frame, that is, the area with smaller pixel values ​​in the final compensation map, the fusion results in the output image using more pixels from the predicted frame.

[0166] In the non-flickering areas of the current original frame, i.e., regions where the compensation map value is close to 1, the output image during fusion almost entirely uses the pixels of the current original frame. In the intermediate regions, i.e., areas with flickering but not severe enough, the content of the current original frame and the predicted frame are mixed proportionally to achieve a smooth transition. This creates a visual persistence effect in the flickering areas of the current original frame, smoothing out flicker changes without affecting the clarity of other areas. The fused output image retains most of the details of the current original frame, while mitigating flickering in the flickering areas by introducing content from the predicted frame.

[0167] After obtaining the output frame, the previous output frame is used as the output frame, and the output image of the scene rendered by the visual engine at the next moment is used as the current original frame. The aforementioned steps 2-6 are repeated to process all frames of the scene rendered by the visual engine. The resulting final output frame composition eliminates or suppresses visual flicker while maintaining the clarity of the image.

[0168] In other words, at the current moment, the present invention obtains an output frame through the above steps. At this time, for the next moment, the frame of the next moment is an image output by the visual engine without any filtering, so it is used as the current original frame; while the output frame obtained through the steps of the present invention is used as the previous output frame, and then the processing of the present invention is repeated multiple times until all images or frames output by the visual engine at all moments are processed in the same way, that is, filtered multiple times, so as to continuously increase the filtering intensity according to the flicker level until the flicker is lower than the set expected value.

[0169] A calculation for compensation, for example Figure 6 As shown, the original frame is the original output image, the predicted frame is the virtual image obtained based on motion prediction, the flicker map shows the flickering pattern obtained according to the algorithm, the compensation map shows the compensation weights obtained from the flicker map, and the output frame is the final image output according to the rules of the compensation map, consisting of the original frame and the predicted frame.

[0170] Alternatively, the present invention can also employ the following alternative strategy in the fusion process: weighted time averaging: This method considers not only the current frame and the previous frame, but also multiple frames of historical data, and weights the historical frames exponentially by time distance, which can more smoothly handle changes in the flickering area.

[0171] Adaptive compensation coefficients are employed: the compensation coefficients are dynamically adjusted based on factors such as scene complexity and motion speed; the contribution of predicted frames is reduced during fast motion to avoid motion blur, while the contribution of predicted frames is increased during stationary or slow motion to enhance flicker suppression; or fusion is based on depth information: if the visual system can provide depth information, it can be combined for more accurate fusion; stronger compensation is applied to distant objects (which are usually more prone to flickering), while nearby objects retain their original details and sharpness. These alternatives can be selected and combined according to the actual application scenario and system performance requirements to further improve the flicker suppression effect.

[0172] As an embodiment of the present invention, the present invention also discloses a visual sawtooth flicker suppression system based on state prediction, the system being used to implement the method, comprising the following modules:

[0173] The acquisition module is used to acquire the image rendered by the visual engine at the current moment as the current raw frame, and the image rendered by the visual engine at the previous moment as the previous output frame.

[0174] The estimation module is used to estimate the transformation relationship between the current original frame and the previous output frame to obtain the transformation matrix;

[0175] The prediction module is used to predict the previous output frame using a transformation matrix and generate a prediction frame.

[0176] The comparison module is used to compare the current original frame with the predicted frame, detect flickering areas, and generate a flickering map;

[0177] The first generation module is used to generate a compensation map based on the flicker map;

[0178] The second generation module is used to fuse the current original frame and the predicted frame using a compensation map to generate the final output frame.

[0179] The continuing processing module is used to repeatedly perform processing, taking the final output frame as the previous output frame and the scene rendered by the visual engine at the next moment as the current original frame.

[0180] As an embodiment of the present invention, the present invention also discloses an electronic device, the electronic device comprising:

[0181] Memory, which stores executable instructions;

[0182] A processor that executes the executable instructions in the memory to implement the method of the present invention.

[0183] As an embodiment of the present invention, the present invention also discloses a computer storage medium storing a computer program, which is executed by a processor to implement the method described in the present invention.

[0184] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” as used in the embodiments of this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0185] The foregoing description illustrates and describes several preferred embodiments of the present invention. However, as previously stated, it should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and environments, and can be altered within the scope of the inventive concept described herein through the foregoing teachings or techniques or knowledge in related fields. Any modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.

Claims

1. A method for suppressing visual sawtooth flicker based on state prediction, characterized in that, The method includes the following steps: S1. Obtain the current frame rendered by the view engine as the current raw frame, and the previous frame rendered by the view engine as the previous output frame. S2. Estimate the transformation relationship between the current original frame and the previous output frame to obtain the transformation matrix, specifically including: S21. Use the ORB algorithm to calculate the key points and descriptors of the current original frame and the previous output frame; S22. Use the BFMatcher algorithm to match feature points in the current original frame and the previous output frame; S23. Based on the similarity between the descriptors of the current original frame and the descriptors of the previous output frame, find the feature point pairs corresponding to the current original frame and the previous output frame. S24. Sort by matching distance and select a certain number of best matching pairs; S25. Perform matching filtering on the best matching pairs to obtain matching feature point pairs; S26. Calculate the transformation matrix based on the matched feature point pairs; S3. Perform prediction on the previous output frame using a transformation matrix to generate a prediction frame, specifically including: S31. For each pixel coordinate in the previous output frame, apply the transformation matrix to calculate the new coordinates of the corresponding pixel in the prediction frame; S32. The inverse matrix of the previous output frame and the transformation matrix is ​​calculated using the interpolation method to obtain the pixel value at the new coordinates of the predicted pixel point; S4. Compare the current original frame with the predicted frame, detect the flickering area, and generate a flicker map, including: S41. Perform grayscale conversion on the current original frame image and the predicted frame image respectively to obtain the grayscale image of the current original frame and the grayscale image of the predicted frame; S42. Calculate the absolute difference between the current original frame grayscale image and the predicted frame grayscale image to obtain the grayscale difference image; S43. If the absolute difference is greater than the set threshold, it is considered that there is a flickering area in the current original frame. The grayscale difference map is subjected to noise suppression processing to obtain the processed flickering map. S5. Generate a compensation map based on the flicker map, including: S51. Remove regions below the preset minimum flicker threshold from the flicker map to obtain a preliminary compensation map; S52. Normalize and filter the preliminary compensation image to obtain a compensation image in which all pixel values ​​are less than or equal to 1; S6. Use the compensation map to fuse the current original frame and the predicted frame to generate the final output frame. Specifically, expand the compensation map to obtain an RGB three-channel compensation map, and use the RGB three-channel compensation map to fuse the current original frame and the predicted frame. S7. Using the final output frame as the previous output frame, and the image rendered by the visual engine at the next moment as the current original frame, repeat steps S2-S6.

2. The method according to claim 1, characterized in that, S1 further includes: processing the current original frame image and the previous output frame image to obtain the same size and convert them to the RGB color space, thereby obtaining the converted current original frame image and the previous output frame image.

3. A visual sawtooth flicker suppression system based on state prediction, characterized in that, The system is used to implement the method according to any one of claims 1-2, and includes the following modules: The acquisition module is used to acquire the image rendered by the visual engine at the current moment as the current raw frame, and the image rendered by the visual engine at the previous moment as the previous output frame. The estimation module is used to estimate the transformation relationship between the current original frame and the previous output frame to obtain the transformation matrix; The prediction module is used to predict the previous output frame using a transformation matrix and generate a prediction frame. The comparison module is used to compare the current original frame with the predicted frame, detect flickering areas, and generate a flickering map; The first generation module is used to generate a compensation map based on the flicker map; The second generation module is used to fuse the current original frame and the predicted frame using a compensation map to generate the final output frame. The continuing processing module is used to repeatedly perform processing, taking the final output frame as the previous output frame and the scene rendered by the visual engine at the next moment as the current original frame.

4. An electronic device, characterized in that, The electronic device includes: Memory, which stores executable instructions; A processor that executes the executable instructions in the memory to implement the method of any one of claims 1-2.

5. A computer storage medium, characterized in that, The medium stores a computer program, which is executed by a processor to implement the method described in any one of claims 1-2.

Images