Image processing methods, electronic devices, computer storage media and computer program products

Abstract

This application provides a screen processing method, an electronic device, a computer storage medium, and a computer program product. The screen processing method includes: during virtual shooting, performing pixel flicker analysis on a screen to be displayed for presenting a virtual scene; determining flickering pixels to be processed in the screen based on the results of the pixel flicker analysis; performing flicker removal processing on the flickering pixels to be processed; and displaying the flicker-removed screen on the screen. This application improves the quality of the screen to be displayed for presenting a virtual scene, thereby improving the quality of the video obtained through virtual shooting.

Description

Technical Field

[0001] This application relates to the field of virtual shooting technology, and in particular to a method for image processing in virtual shooting, an electronic device, a computer storage medium, and a computer program product. Background Technology

[0002] With the rapid development of VR (Virtual Reality), AR (Augmented Reality), and MR (Mixed Reality) technologies, virtual shooting, as an emerging film and television production method that integrates multiple advanced technologies, has been widely used in film, television, games, advertising, and other fields.

[0003] Virtual filming is a technique that uses a virtual scene as the background and a real object as the foreground for shooting. One major method in this technique is using a screen as an intermediary. For example, LED virtual filming technology, which uses an LED (Light Emitting Diode) screen as an intermediary, utilizes a virtual scene projected onto a high-resolution LED screen as the background. Combined with real-time image processing technology, it integrates the virtual scene with the real-world shooting. In LED virtual filming, assuming the real-world object in the foreground is an actor, the actor is placed in front of the virtual scene projected onto the LED screen. Through computer graphics processing, the actor's movements and expressions can be blended with the virtual scene, making the shooting background more realistic. Other screen-mediated virtual filming techniques are implemented similarly to LED virtual filming.

[0004] However, in actual virtual shooting processes, the screen used to present the virtual scene, i.e., the background, often experiences screen flickering due to various reasons. This phenomenon not only affects the quality of the captured footage but also the quality of the final virtual video, negatively impacting the viewer's experience. Summary of the Invention

[0005] In view of this, embodiments of this application provide a screen processing solution to at least partially solve the above-mentioned problems.

[0006] According to a first aspect of the embodiments of this application, a screen processing method is provided, including:

[0007] During virtual shooting, pixel flicker analysis is performed on the image to be presented on the screen used to present the virtual scene; based on the result of the pixel flicker analysis, flickering pixels to be processed in the image are identified; flicker removal processing is performed on the flickering pixels to be processed, and the image after flicker removal processing is presented on the screen.

[0008] According to a second aspect of the present application, an electronic device is provided, including: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other through the communication bus; the memory is used to store at least one executable instruction, which causes the processor to perform an operation corresponding to the method described in the first aspect.

[0009] According to a third aspect of the embodiments of this application, a computer storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.

[0010] According to a fourth aspect of the embodiments of this application, a computer program product is provided, including computer instructions that instruct a computing device to perform an operation corresponding to the method described in the first aspect.

[0011] According to the solution provided in this application, in the method of presenting a virtual scene through a screen, or projecting a virtual scene as a shooting background using a screen as an intermediary to achieve virtual shooting, pixel flicker analysis is performed on the image to be presented on the screen used to present the virtual scene. Based on the result of the pixel flicker analysis, flickering pixels in the image are identified and then flicker removal processing is performed. Thus, the flickering pixels are processed before the image is presented on the screen, resulting in a flicker-removed image, meaning there are no flickering pixels in the image, but rather a relatively high-quality, stable, and well-presented image. Based on this, the video obtained by virtual shooting based on the image presented on the screen is of higher quality.

[0012] Compared to traditional methods that use hardware to remove flicker, the solution in this application does not require additional hardware or modifications to the hardware. This not only improves the quality of the presented image but also reduces the implementation cost of flicker removal. Compared to traditional methods that use complex rendering algorithms to remove flicker, the solution in this application is simpler, more efficient, and more versatile. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0014] Figure 1A This is a schematic diagram of an exemplary system to which the embodiments of this application are applicable;

[0015] Figure 1B This is a schematic diagram of another exemplary system to which the embodiments of this application are applicable;

[0016] Figure 2 This is a flowchart illustrating the steps of a screen processing method according to an embodiment of this application.

[0017] Figure 3A This is a schematic diagram of the rendering trajectory of a pixel in a two-dimensional coordinate system according to an embodiment of this application;

[0018] Figure 3B This is a schematic diagram of the rendering trajectory of a second type of pixel according to an embodiment of this application in a two-dimensional coordinate system;

[0019] Figure 3C This is a schematic diagram of the rendering trajectory of a third pixel according to an embodiment of this application in a two-dimensional coordinate system;

[0020] Figure 4 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0021] To enable those skilled in the art to better understand the technical solutions in the embodiments of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art should fall within the protection scope of the embodiments of this application.

[0022] The following, combined with Figure 1A and Figure 1B The present application describes virtual shooting scenes and exemplary systems applicable to the embodiments of this application.

[0023] As mentioned earlier, virtual filming technology involves both "real" and "virtual." The "real" aspect refers to the physical screens and cameras in the actual environment during virtual filming. The "virtual" aspect refers to the virtual scenes constructed based on filming needs. These virtual scenes can be presented through physical screens, or projected onto surfaces such as screens or walls using a physical screen as an intermediary. During virtual filming, actors in the physical environment can perform in front of physical screens, screens, or walls displaying virtual backgrounds, while being filmed by physical cameras, thus achieving a combination of "virtual" and "real."

[0024] For example, refer to Figure 1A An exemplary system applicable to embodiments of this application is shown. For example... Figure 1AAs shown, the system 100A may include a physical camera 102A and a screen 104A for presenting a virtual scene. The screen 104A displays the pre-constructed virtual scene from the shooting perspective of the virtual camera. Those skilled in the art should understand that the positional relationship between the physical camera 102A and the screen 104A is not limited in this embodiment; the physical camera 102A only needs to capture a good view of the actual object and a complete or partial area of ​​the virtual scene presented on the screen 104A. When an actual object, such as an actor, performs actions in front of the screen 104A, the physical camera 102A captures the virtual scene presented on the screen 104A as a background, achieving effective fusion of "virtual" and "real" elements to create an effect similar to an actual object moving in a real scene. For example, the screen 104A in this example can be an LED screen.

[0025] Regarding the screen 104A, which displays a virtual scene, pixel flickering may occur when displaying the virtual scene. This flickering can be captured by the physical camera 102A in the video frame, resulting in poor video quality. Therefore, in this embodiment, flickering pixels in the image to be displayed are processed before the virtual scene is shown on the screen 104A, ensuring that the image displayed on the screen 104A is flicker-free. In other words, flickering pixels are eliminated, resulting in higher quality video frames obtained by the physical camera 102A from the image displayed on the screen 104A.

[0026] However, this is not the only example; in another feasible exemplary system that applies the embodiments of this application, such as... Figure 1B As shown, the system 100B includes: a physical camera 102B, a screen 104B for projecting a virtual scene, and a foreground curtain 106B. The foreground curtain 106B can be located at any position capable of receiving the virtual scene projected by the screen 104B, allowing for better adaptation to more complex shooting environments. Those skilled in the art should understand that in this embodiment, "foreground curtain" refers to any medium capable of presenting a virtual scene. In practical applications, this includes, but is not limited to, the form of a curtain, and can also be other forms, such as a wall or a large screen, all of which are applicable to the scheme of this embodiment. The positional relationship between the physical camera 102B and the foreground curtain 106B is not limited; the physical camera 102B only needs to be able to capture a good image of the actual object and the complete or partial area of ​​the virtual scene presented by the foreground curtain 106B. When an actual object, such as an actor, moves in front of the foreground curtain 106B, with the virtual scene presented by the foreground curtain 106B as the background, the physical camera 102B captures the image, achieving effective "virtual" and "real" fusion to create an effect similar to the actual object moving in a real scene.

[0027] In this example, the virtual scene displayed on screen 104B, to be projected onto foreground screen 106B, may exhibit pixel flickering. Consequently, the virtual scene displayed on foreground screen 106B may also exhibit pixel flickering, causing physical camera 102B to capture this flickering in the video frame, resulting in poor video quality. Therefore, in this embodiment, before the virtual scene is displayed on screen 104B, the flickering pixels in the displayed image are processed, ensuring that the image displayed on screen 104B is flicker-free. Screen 104B then projects the flicker-free virtual scene onto foreground screen 106B. That is, there are no flickering pixels in the image, and the video frame obtained by physical camera 102B from the virtual shot of the image displayed on foreground screen 106B is of higher quality.

[0028] Based on the above system, the solutions provided in the embodiments of this application will be described below through multiple examples.

[0029] Reference Figure 2 The diagram illustrates a flowchart of a screen processing method according to an embodiment of this application.

[0030] The image processing method in this embodiment is used for virtual shooting scenes, and includes the following steps:

[0031] Step S202. During the virtual shooting process, perform pixel flicker analysis on the image to be presented on the screen used to present the virtual scene.

[0032] In this embodiment, the screen used to present the virtual scene can be any suitable screen capable of rendering and presenting images, for example, it can be such as Figure 1A The screen 104A shown, or as shown Figure 1B Screen 104B, etc., shown in the figure.

[0033] Whether using the screen directly as a background or projecting the image onto other media, the image needs to be displayed on the screen, and the foundation of this display is pixels. However, due to hardware or software limitations, some pixels often flicker, affecting the image quality and ultimately impacting the overall quality of the virtual video. Therefore, it's necessary to identify and remove flickering pixels to improve image quality.

[0034] Therefore, this step performs pixel flicker analysis on a frame before it is displayed on the screen.

[0035] Taking Unreal Engine (UE) for rendering as an example, the process begins by loading models and materials to construct the virtual scene, and then setting the position of the virtual camera to frame the scene. The virtual camera is a device that simulates the working principle of a real camera, capable of capturing objects within the virtual scene. After setting the virtual camera's focal length and other parameters, and determining its position and orientation, the rendering virtual scene can be previewed in real-time through the camera's viewpoint during UE operation. Further, objects outside the current viewpoint are removed; then, the visibility of objects within the current viewpoint is assessed, and invisible objects are removed—that is, objects occluded within the current viewpoint are removed. In an image processor, a 3D (Three Dimensions) coordinate system is mapped to a 2D (Two Dimensions) coordinate system, transforming the original 3D scene data into pixels visible on the screen. This converts information such as objects and lighting in a pre-rendered 3D virtual scene into multiple frames of 2D images at a specific time sequence, allowing viewers to experience 3D visual effects within a 2D context. In practical applications, virtual scenes can be pre-rendered using rendering engines such as UE, Unity, and Arnold for flexible use during virtual filming. Before being displayed on the screen used to present the virtual scene, the rendering engine performs pixel flicker analysis on each frame of the converted 2D images.

[0036] To address pixel flicker, the solution in this application's embodiments performs pixel flicker analysis on the frame of the image to be presented from multiple angles. In practical applications, some angles or all angles can be used.

[0037] In one scenario, during virtual photography, rapid movement or shaking of the subject can cause flickering of pixels in the rendered 2D image. Alternatively, if there are moving objects in the virtual scene, improper virtual camera settings, such as shutter speed or motion blur, can result in poor motion blur effects from the rendering engine. For example, if the virtual camera's shutter speed is slow but there are fast-moving objects in the virtual scene, the pixels constituting the moving objects may exhibit irregular movement in the 2D image converted from the virtual scene, causing pixel flickering. Furthermore, if the anti-aliasing parameters or algorithm are improperly set during rendering of the virtual scene—for instance, if the anti-aliasing sample count is set too low—the edges of the image may appear jagged in the 2D image converted from the virtual scene, causing pixel flickering.

[0038] In other words, pixel flickering will occur in the image to be displayed, which is related to the motion information of the pixels. Based on this, pixel flicker analysis of the image to be displayed can be implemented by performing motion analysis on the image to detect pixel flicker. For example, based on the motion information of each pixel, it can be determined whether there are any pixels in the image to be displayed that flicker due to changes in motion information during the sequential display process. In one example, the motion information can be the velocity vector of the pixel.

[0039] From another perspective, when pre-rendering virtual scenes using a rendering engine, PDO (Pixel Depth Offset) technology is needed to handle the transitions between multiple models or between a model and terrain to achieve a natural transition effect. Specifically, when rendering a virtual scene, PDO technology changes the perceived distance between pixels and the virtual camera; that is, it adjusts the details and occlusion relationships of models or terrain at their contact points with other objects by changing the pixel's depth information. This causes pixels that were originally at the same depth to appear visually different, achieving a natural transition effect. In practical applications, a depth buffer records the depth information of each pixel in the virtual scene to determine which pixels should be displayed and which should be occluded during rendering. When the depth information of two objects' pixels is very close, and the depth buffer, due to precision limitations, cannot accurately distinguish the display or occlusion relationship between the two objects, depth conflicts may occur between them. Pixels with similar depth information are affected and thus alternate between different frames in the rendered virtual scene. In other words, two pixels with different depth information alternate in the same position in the frame, causing pixel flickering. In another scenario, if the PDO setting of a pixel is not set properly, some pixels may be incorrectly rendered in front or occluded. Due to the change in depth information, the incorrectly rendered pixels may appear alternately between different frames, resulting in pixel flickering.

[0040] In other words, pixel flickering related to the depth information of pixels will occur in the image to be displayed. Based on this, pixel flickering analysis of the image to be displayed can be implemented by performing depth analysis on the image to detect pixel flickering. For example, based on the depth information of each pixel, it can be determined whether there are pixels whose depth information changes during the sequential display of the image to be displayed, thus causing flickering in the image to be displayed. In one example, the depth information can be the depth value of the pixel.

[0041] From another perspective, light sources are a key factor in constructing virtual scenes, providing illumination and visual guidance. In rendering engines, lighting calculations are used to obtain the lighting conditions on the surfaces of objects. If the light source is unstable, or if the lighting algorithm has defects in the setting of light source parameters or object materials, abnormal changes in the color information of pixels in the virtual scene may occur during rendering. For example, sudden changes in pixel color or irregular fluctuations in pixel brightness may occur, resulting in a visually flickering pixel phenomenon.

[0042] In other words, pixel flickering related to the color information of pixels will occur in the image to be displayed. Based on this, pixel flicker analysis of the image to be displayed can be implemented by performing color analysis on the image to achieve pixel flicker detection. For example, based on the color information of each pixel, it can be determined whether there are pixels whose color information changes and causes flickering in the image to be displayed during the sequential presentation of the image. In one example, the color information can be the pixel color value represented by the red-green-blue color model.

[0043] In summary, in one example of the embodiments of this application, image-based pixel flicker analysis may include at least one of the following: motion analysis of the image, depth analysis of the image, and color analysis of the image. Any one of these analyses can achieve the effect of detecting flickering pixels, but using multiple analyses simultaneously can achieve more comprehensive pixel flicker detection and more accurate detection results.

[0044] The following sections will explain the various pixel flicker analyses described above.

[0045] (I) Motion analysis of the image

[0046] When pixel flicker analysis includes motion analysis, motion information of each pixel in the image to be presented can be obtained. This allows for further verification, using motion information, of whether flickering pixels exist in the virtual scene due to irregular pixel movement. Specifically, in one feasible approach, motion information of pixels in the image to be presented can be determined based on historical images that are temporally adjacent to the image to be presented. This motion information reflects the movement of pixels from their positions in historical images to their positions in the image to be presented.

[0047] For a given object in an image, its movement or the change in shooting angle will exhibit certain patterns. Correspondingly, the motion information of its pixels will also reflect these patterns. If the motion information of a pixel does not conform to its corresponding motion pattern, then that pixel can be considered to be flickering.

[0048] In one example, at least one historical frame that is sequentially adjacent to the frame to be presented is obtained; the pixels in the at least one historical frame that correspond to the pixels in the frame to be presented are determined, along with the motion information of the corresponding pixels; and based on the motion information of the corresponding pixels, the motion information of the pixels in the frame to be presented is determined.

[0049] During the process of each pixel moving from its position in a historical frame to its position in the frame to be presented, the change in pixel motion state between temporally adjacent frames is relatively small under normal circumstances. Therefore, when acquiring the motion information of pixels in the frame to be presented, at least one historical frame temporally adjacent to the frame to be presented can be referenced to obtain more accurate motion information. The number of historical frames acquired can be set according to actual needs, for example, based on the frame rate when Unreal Engine renders the virtual scene, i.e., the number of frames generated per second: the number of historical frames temporally adjacent to the frame to be presented is set to the number of frames generated per second based on the current frame rate. However, this is not limited to this; other setting methods, such as setting based on data statistics results or experimental simulation results, are also applicable to the solutions in this application embodiment.

[0050] Based on at least one historical frame that is temporally adjacent to the frame to be presented, the pixels corresponding to the pixels in the frame to be presented, and their motion information, are determined in each historical frame to confirm the temporal motion information of the pixels in the frame to be presented. For example, in 10 consecutive historical frames, a coin exists. As the frame progresses, the coin's motion information changes; from a pixel perspective, this means the motion information of the pixels corresponding to the coin changes. Since these pixels all correspond to the object "coin," although their motion information changes, the correspondence of a pixel across multiple frames and the overall change in its motion information can be determined based on pixel similarity between adjacent frames. Based on this, accurate motion information is determined for the pixels corresponding to that pixel in the frame to be presented (e.g., frame 11).

[0051] Based on this, in one feasible approach, pixels in historical images that correspond to pixels in the image to be presented can be identified based on the similarity between pixels.

[0052] In one example, the position of a pixel in the image to be presented is determined, and a pixel region of a preset size containing that pixel is determined based on the position; from the region corresponding to the pixel region in at least one historical frame, the pixel corresponding to the pixel in the image to be presented is determined.

[0053] In one alternative approach, for a pixel in the image to be presented, its position is determined within the image. Based on this position, a pixel region of a preset size including the pixel is identified within the image. Then, pixels similar to the pixel are determined from the corresponding pixel region in historical images. In a multi-frame virtual scene, frames that are temporally adjacent or close in time along the timeline have continuity. The change in pixel motion state between two temporally adjacent frames is relatively small, thus avoiding changes that excessively affect the user's visual experience in consecutive frames. If a pixel moves within a frame that is temporally adjacent or close in time along the timeline, its position in the frame is similar, and the range of pixel movement is within a certain area. Therefore, when identifying a pixel in at least one historical frame that corresponds to a pixel in the image to be presented, the identification is based on a pixel region of a preset size including the pixel in the image to be presented. For example, a first pixel region of a preset size including pixel A is identified in the image to be presented. This first pixel region is a square region with a side length of five pixels, centered on pixel A identified in the image to be presented.

[0054] In practical applications, the preset size of the first pixel region can be adjusted based on the resolution or size of the image to be presented, and the distance of the confirmed historical frames from the image to be presented on the timeline. For example, the larger the resolution or size of the image to be presented, the greater the range of movement of the pixels in the image to be presented within the confirmed historical frames. Therefore, the range of the first pixel region is correspondingly larger, in order to more accurately identify the pixels corresponding to the pixels in the image to be presented. For another example, when there are many confirmed historical frames, the first pixel region is set separately for each historical frame based on their distance from the image to be presented on the timeline. In one example, when 100 historical frames have been confirmed, for a certain pixel in the image to be presented (such as frame 101), that pixel moves from its position in frame 1 through its positions in frames 2, 3, 4, ..., 99, 100, finally reaching its position in the image to be presented. When a pixel moves from a position in a historical frame farther away from the frame to be presented, it undergoes more movements to reach its current position in the frame to be presented. Therefore, the pixel may experience a larger displacement, and a larger first pixel area can be set in the historical frame accordingly. Conversely, when a pixel moves from a position in a historical frame closer to the frame to be presented, it undergoes fewer movements to reach its current position in the frame to be presented. Therefore, the pixel may experience a smaller displacement, and a smaller first pixel area can be set in the historical frame accordingly. Based on this principle, in the historical frames 1 to 30, the first pixel area is a square area with a side length of 6 pixels, centered on pixel A in the frame to be presented; in the historical frames 31 to 60, the first pixel area is a square area with a side length of 4 pixels, centered on pixel A in the frame to be presented; and in the historical frames 61 to 100, the first pixel area is a square area with a side length of 2 pixels, centered on pixel A in the frame to be presented. Therefore, based on the magnitude of the possible displacement of pixels in actual situations, a first pixel region is set for different historical images, which increases the accuracy of confirming the pixels corresponding to the pixels in the image to be presented.

[0055] After identifying a first pixel region of a preset size, including pixel A, in the image to be presented, a second pixel region corresponding to the first pixel region is identified in at least one historical frame. Since the change in pixel motion state between two temporally adjacent frames is small, identifying the pixel corresponding to the pixel in the image to be presented within the second pixel region of at least one historical frame improves the accuracy of obtaining the pixel corresponding to the pixel in the image to be presented. Specifically, in one example, the relative position of the second pixel region in the historical frame is the same as the relative position of the first pixel region in the image to be presented, and the second pixel region is the same size as the first pixel region, so as to perform accurate pixel comparison within the second pixel region, thereby identifying pixel A' corresponding to pixel A in the image to be presented within the second pixel region. Furthermore, all pixels in the second pixel region are traversed, and the similarity between each pixel and pixel A in the image to be presented is calculated. For example, the Euclidean distance and cosine similarity between each pixel and pixel A in the image to be presented are calculated. The pixel in the second pixel region with the highest similarity to pixel A in the image to be presented is selected as the pixel A' corresponding to pixel A in the image to be presented, and the motion information of the corresponding pixel A' is obtained accordingly. Thus, by using the above method, the pixel corresponding to the pixel in the image to be presented is identified in each acquired historical frame, without having to identify it in the entire historical frame, reducing the amount of computation.

[0056] After confirming the pixels in at least one historical frame that correspond to the pixels in the image to be presented, the motion information of the pixels in the image to be presented is determined based on the motion information of the pixels in at least one historical frame that correspond to the pixels in the image to be presented. In this way, the motion of each pixel in the image to be presented is obtained more comprehensively, providing a basis for judging whether there are flickering pixels in the image to be presented caused by irregular pixel motion based on the motion information.

[0057] The motion information of the pixel corresponding to the pixel in the image to be presented in at least one historical frame can be represented as follows: the motion state of the pixel when the corresponding pixel moves from its position in the previous historical frame to its position in the current historical frame. In one feasible approach, the motion information of the pixel corresponding to the pixel in the current historical frame can be implemented as the velocity vector of the pixel when it moves from its position in the previous historical frame to its position in the current historical frame. This velocity vector can be used to describe the magnitude and direction of the pixel's motion. For example, for 10 confirmed historical frames, the motion information of the pixel corresponding to the pixel in the current historical frame in the 10th historical frame can be represented as the velocity vector of the pixel when it moves from its position in the 9th historical frame to its position in the 10th historical frame. This velocity vector is a two-dimensional vector containing the magnitude and direction of the pixel's velocity along the x-axis and y-axis.

[0058] In one feasible approach, the motion information of pixels in at least one historical frame can be obtained using the method described in this embodiment, which involves acquiring the motion information of pixels in the frame to be presented. That is, each historical frame in at least one historical frame is used as the frame to be presented, and the motion information of pixels in the frame to be presented is obtained. Thus, the motion information of pixels in at least one historical frame is obtained, and after determining the pixels in at least one historical frame that correspond to the pixels in the frame to be presented, the motion information of the corresponding pixels is determined. In another feasible approach, for the first frame of a virtual scene in time sequence, since this frame does not have historical frames, the motion information of pixels in the first frame of the virtual scene in time sequence that corresponds to the pixels in the frame to be presented can be preset before confirming the motion information of pixels in the first frame of the virtual scene in time sequence. The motion information of pixels in the first frame of the virtual scene in time sequence can be obtained based on the experience of those skilled in the art or through prediction using models such as machine learning models. However, this is not the only one. In practical applications, if the rendering engine of the virtual scene is connected to an external data source, such as a motion capture system, the motion information of the pixels in the first frame of the sequence can be set according to the external data source.

[0059] In one feasible approach, the motion information of pixels corresponding to pixels in at least one historical frame that are to be presented is weighted according to their temporal relationship with the image to be presented and a first preset weight. For example, the motion information of pixels corresponding to pixels in at least one historical frame that are to be presented is weighted according to their temporal relationship with the image to be presented and a first preset weight; based on the weighting result, the motion information of pixels in the image to be presented is determined.

[0060] The first preset weight can be appropriately set by those skilled in the art according to actual needs. For example, based on historical experience or statistical data, according to the temporal relationship with the image to be presented, for historical images that are closer to the image to be presented on the timeline, the change in the motion state of the pixels corresponding to the pixels in the image to be presented in the historical image is smaller, and the reference value of the motion information of the corresponding pixels is greater, so a slightly heavier weight can be set. However, it is not limited to this; the weight can also be evenly distributed for at least one historical image, etc. In one feasible method, based on the above experience, for example, for three historical images (frames 1, 2, and 3) that are temporally adjacent to the image to be presented (frame 4), according to their distance from the image to be presented on the timeline from farthest to closest, the motion information of the corresponding pixels in the three historical images is set with weights of 0.2 (frame 1 weight), 0.3 (frame 2 weight), and 0.5 (frame 3 weight). Assuming the motion information of pixel A in frames 1, 2, and 3 corresponding to pixel A in frame 4 is V1, V2, and V3 respectively, we can obtain the motion information of pixel A in frame 4 as: 0.2V1 + 0.3V2 + 0.5V3. That is, the weighted processing result of pixels in frames 1, 2, and 3 is used as the motion information of pixel A in frame 4.

[0061] Therefore, by assigning weights to the motion information of corresponding pixels in different historical images based on their reference value to the images to be presented, the motion information of pixels in the images to be presented can be made more objective and accurate.

[0062] (II) In-depth analysis of the image

[0063] When pixel flicker analysis includes depth analysis, it is based on the same principle as the aforementioned method for confirming the motion information of pixels in the image to be presented. It can obtain the depth information of each pixel in the image to be presented, so as to further confirm whether there are flickering pixels in the virtual scene image caused by changes in the depth information of pixels.

[0064] In a virtual scene, once the positional relationship between an object and the virtual camera is confirmed, the depth information of its pixels is also confirmed. As the positional relationship between an object and the virtual camera changes, the depth information of its pixels will exhibit a certain pattern of change. If the depth information of a pixel does not conform to this pattern of change, it can be considered that the pixel is flickering.

[0065] In one feasible approach, at least one historical frame that is sequentially adjacent to the frame to be presented is acquired; the depth information of the pixels in the at least one historical frame that correspond to the pixels in the frame to be presented is determined; and the depth information of the pixels in the frame to be presented is determined based on the depth information of the corresponding pixels.

[0066] The specific implementation method for obtaining at least one historical frame that is temporally adjacent to the frame to be presented can be referred to in (I) when performing motion analysis on the frame, which is the specific implementation method for obtaining at least one historical frame that is temporally adjacent to the frame to be presented. It will not be repeated here.

[0067] In at least one historical frame adjacent in time to the frame to be presented, the change in pixel depth information is relatively small under normal circumstances. Therefore, in one feasible approach, when acquiring the depth information of pixels in the frame to be presented, at least one historical frame adjacent in time to the frame to be presented can be referenced to obtain more accurate depth information. This allows for a more comprehensive acquisition of the depth information of each pixel in the frame to be presented, providing a basis for determining whether there are flickering pixels in the frame to be presented due to changes in pixel depth information. The number of historical frames acquired can be set according to actual needs, for example, based on the frame rate when Unreal Engine renders a virtual scene, i.e., the number of frames generated per second: the number of historical frames adjacent in time to the frame to be presented is set to the number of frames generated per second based on the current frame rate. However, this is not limited to this; other setting methods, such as setting based on data statistics or experimental simulation results, are also applicable to the solutions in this application embodiment.

[0068] Based on at least one historical frame temporally adjacent to the image to be presented, the pixels corresponding to the pixels in the image to be presented, and their depth information, are determined in each historical frame to confirm the temporal changes in the depth information of the pixels in the image to be presented. For example, in 10 consecutive historical frames, a sphere exists, and the distance between the sphere and the virtual camera gradually decreases. From a pixel perspective, this means the depth information of the pixels corresponding to the sphere has changed. Since these pixels all correspond to the object "sphere," although their depth information has changed, the correspondence of a pixel across multiple frames and the overall change in its depth information can be determined based on pixel similarity between adjacent frames. Based on this, accurate depth information is determined for the pixels corresponding to that pixel in the image to be presented (e.g., the 11th frame).

[0069] Based on this, in one feasible approach, pixels in historical images that correspond to pixels in the image to be presented can be identified based on the similarity between pixels.

[0070] In one example, the position of a pixel in the image to be presented is determined, and a pixel region of a preset size containing that pixel is determined based on the position; from the region corresponding to the pixel region in at least one historical frame, the pixel corresponding to the pixel in the image to be presented is determined.

[0071] In one alternative approach, for a pixel in the image to be presented, its position is determined within the image. Based on this position, a pixel region of a preset size including the pixel is identified within the image. Then, pixels similar to the pixel are determined from the corresponding pixel region in historical images. In a multi-frame virtual scene, frames that are temporally adjacent or close in time along the timeline have continuity. The change in pixel depth information between two temporally adjacent frames is small, thus avoiding changes that excessively impact the user's visual experience in consecutive frames. If pixel depth information changes in temporally adjacent or close in time along the timeline, the pixel's position in the frame is similar, and the change in the relative position between the pixel and the virtual camera is small, with the change in the pixel's position within a certain area. Therefore, when identifying pixels in at least one historical frame that correspond to a pixel in the image to be presented, confirmation is based on a pixel region of a preset size including the pixel in the image to be presented, eliminating the need for confirmation across the entire historical frame and reducing computational load.

[0072] The specific implementation method for confirming the pixel corresponding to the pixel in the image to be presented in at least one historical frame can be referred to in (I) when performing motion analysis on the image to confirm the specific implementation method of the corresponding pixel, which will not be repeated here.

[0073] Based on this, after confirming the pixels in at least one historical frame that correspond to the pixels in the image to be presented, the depth information of the pixels in the image to be presented is determined according to the depth information of the pixels in at least one historical frame that correspond to the pixels in the image to be presented. This allows for a more comprehensive acquisition of the depth information of each pixel in the image to be presented, providing a basis for determining whether there are flickering pixels in the image to be presented due to changes in depth information.

[0074] In one feasible approach, the depth information of pixels in at least one historical frame can be obtained by acquiring the depth information of pixels in the frame to be presented, as described in this embodiment. That is, each historical frame in at least one historical frame is used as the frame to be presented, and the depth information of pixels in the frame to be presented is acquired. Thus, the depth information of pixels in at least one historical frame is obtained, and after determining the pixels in at least one historical frame that correspond to the pixels in the frame to be presented, the depth information of the corresponding pixels is determined. In another feasible approach, for the first frame of a virtual scene in time sequence, since this frame does not have historical frames, the first frame can be analyzed using image processing algorithms, feature extraction algorithms, etc., and the depth information of pixels in at least one historical frame can be obtained based on the analysis results.

[0075] In one feasible approach, the depth information of pixels corresponding to pixels in the image to be presented in at least one historical frame is weighted according to their temporal relationship with the image to be presented and a second preset weight. For example, the depth information of pixels corresponding to pixels in the image to be presented in at least one historical frame is weighted according to their temporal relationship with the image to be presented and a second preset weight; the depth information of pixels in the image to be presented is determined based on the weighting result.

[0076] The second preset weight can be appropriately set by those skilled in the art according to actual needs. For example, it can be set to the same as the first preset weight, or it can be based on historical experience or statistical data, according to the temporal relationship with the image to be presented. For historical images that are closer to the image to be presented on the timeline, the depth information of the corresponding pixels in the historical images has changed less, and the reference value of the depth information of the corresponding pixels in the image to be presented is greater, so a slightly heavier weight can be set. However, it is not limited to this; the weight can also be evenly distributed among at least one historical image, etc. By setting weights for the depth information of corresponding pixels in different historical images according to the reference value of different historical images to the image to be presented, the depth information of each pixel in the image to be presented is made more objective and accurate.

[0077] The specific implementation method for weighting the depth information of pixels corresponding to pixels in the image to be presented in at least one historical frame can be found in (I) when performing motion analysis on the image, which is the specific implementation method for weighting the motion information of pixels corresponding to pixels in the image to be presented in at least one historical frame. It will not be repeated here.

[0078] (III) Color analysis of the image

[0079] When pixel flicker analysis includes color analysis, it is based on the same principle as the aforementioned method for confirming the motion information of pixels in the image to be presented. It can obtain the color information of each pixel in the image to be presented, so as to further confirm whether there are flickering pixels in the virtual scene image caused by changes in the color information of pixels.

[0080] For a given object in the image, as the position or intensity of the light source in the virtual scene changes, the lighting conditions it receives will exhibit certain patterns. Correspondingly, the color information of the pixels in the area affected by the lighting will also exhibit these patterns. If the color information of a pixel does not conform to these patterns, it can be considered that the pixel is flickering.

[0081] In one feasible approach, at least one historical frame that is sequentially adjacent to the frame to be presented is acquired; the color information of the pixels in the at least one historical frame that correspond to the pixels in the frame to be presented is determined; and the color information of the pixels in the frame to be presented is determined based on the color information of the corresponding pixels.

[0082] The specific implementation method for obtaining at least one historical frame that is temporally adjacent to the frame to be presented can be referred to in (I) when performing motion analysis on the frame, which is the specific implementation method for obtaining at least one historical frame that is temporally adjacent to the frame to be presented. It will not be repeated here.

[0083] In at least one historical frame adjacent to the frame to be presented, the color information of pixels typically changes relatively little. Therefore, in one feasible approach, when acquiring the color information of pixels in the frame to be presented, at least one historical frame adjacent to the frame to be presented can be referenced to obtain more accurate color information. This allows for a more comprehensive acquisition of the color information of each pixel in the frame to be presented, providing a basis for determining whether there are flickering pixels in the frame to be presented due to changes in pixel color information. The number of historical frames acquired can be set according to actual needs, for example, based on the frame rate when Unreal Engine renders a virtual scene, i.e., the number of frames generated per second: the number of historical frames adjacent to the frame to be presented is set to the number of frames generated per second based on the current frame rate. However, this is not limited to this; other setting methods, such as setting based on data statistics or experimental simulation results, are also applicable to the solutions in this application embodiment.

[0084] Based on at least one historical frame sequentially adjacent to the image to be presented, the pixels corresponding to the pixels in the image to be presented in each historical frame, and their color information, are determined to confirm the temporal changes in color information of the pixels in the image to be presented. For example, in 10 consecutive historical frames, there exists a sphere illuminated by a fixed light source, the color of which switches between white and yellow at a fixed frequency. From a pixel perspective, this means the color information of the pixels corresponding to the sphere has changed. Since these pixels all correspond to the object "sphere," although their color information has changed, the correspondence of a pixel across multiple frames and the overall change in its color information can be determined based on pixel similarity between adjacent frames. Based on this, accurate color information is determined for the pixels corresponding to that pixel in the image to be presented (e.g., frame 11).

[0085] Based on this, in one feasible approach, pixels in historical images that correspond to pixels in the image to be presented can be identified based on the similarity between pixels.

[0086] In one example, the position of a pixel in the image to be presented is determined, and a pixel region of a preset size containing that pixel is determined based on the position; from the region corresponding to the pixel region in at least one historical frame, the pixel corresponding to the pixel in the image to be presented is determined.

[0087] In one alternative approach, for a pixel in the image to be presented, its position is determined within the image. Based on this position, a pixel region of a preset size including the pixel is identified within the image. Then, pixels similar to the pixel are determined from the corresponding pixel region in historical images. In a multi-frame virtual scene, frames that are temporally adjacent or close in time along the timeline have continuity. If the position of a pixel changes between two temporally adjacent frames, the change is minimal to avoid creating excessive changes that negatively impact the user's visual experience in consecutive frames. Therefore, when identifying pixels in at least one historical frame that correspond to a pixel in the image to be presented, confirmation is based on a pixel region of a preset size including the pixel in the image to be presented, eliminating the need for confirmation across the entire historical frame and reducing computational load.

[0088] The specific implementation method for confirming the pixel corresponding to the pixel in the image to be presented in at least one historical frame can be referred to in (I) when performing motion analysis on the image to confirm the specific implementation method of the corresponding pixel, which will not be repeated here.

[0089] Based on this, after confirming the pixels in at least one historical frame that correspond to the pixels in the image to be presented, the color information of the pixels in the image to be presented is determined according to the color information of the pixels in at least one historical frame that correspond to the pixels in the image to be presented. This provides a more comprehensive acquisition of the color information of each pixel in the image to be presented, and provides a basis for judging whether there are flickering pixels in the image to be presented due to changes in color information.

[0090] In one feasible approach, the color information of pixels in at least one historical frame can be obtained using the method described in this embodiment, which involves obtaining the color information of pixels in the image to be presented. That is, each historical frame in at least one historical frame is used as the image to be presented, and the color information of pixels in the image to be presented is obtained. Thus, the color information of pixels in at least one historical frame is obtained, and after determining the pixels in at least one historical frame that correspond to the pixels in the image to be presented, the color information of the corresponding pixels is determined. In another feasible approach, for the first frame of a virtual scene in time sequence, since this frame does not have historical frames, the first frame can be analyzed using image processing algorithms, feature extraction algorithms, etc., and the color information of pixels in at least one historical frame can be obtained based on the analysis results.

[0091] In one feasible approach, the color information of pixels corresponding to pixels in the image to be presented in at least one historical frame is weighted according to their temporal relationship with the image to be presented and a third preset weight. For example, the color information of pixels corresponding to pixels in the image to be presented in at least one historical frame is weighted according to their temporal relationship with the image to be presented and a third preset weight; the color information of the pixels in the image to be presented is determined based on the weighting result.

[0092] The third preset weight can be appropriately set by those skilled in the art according to actual needs. For example, it can be set to be the same as the first or second preset weight, or it can be based on historical experience or statistical data, according to the temporal relationship with the image to be presented. For historical images that are closer to the image to be presented on the timeline, the color information of the corresponding pixels in the historical images has changed less, and the reference value of the color information of the corresponding pixels in the image to be presented is greater, so a slightly heavier weight can be set. However, it is not limited to this; the weight can also be evenly distributed among at least one historical image, etc. By setting weights for the color information of corresponding pixels in different historical images according to their reference value to the image to be presented, the color information of each pixel in the image to be presented becomes more objective and accurate.

[0093] The specific implementation method for weighting the color information of pixels corresponding to pixels in the image to be presented in at least one historical frame can be found in (I) when performing motion analysis on the image, which is the specific implementation method for weighting the motion information of pixels corresponding to pixels in the image to be presented in at least one historical frame. It will not be repeated here.

[0094] In one example, before acquiring at least one historical frame that is temporally adjacent to the frame to be presented, at least one of the following is determined: whether there are pixels without motion information in the frame to be presented; whether there are pixels without depth information in the frame to be presented; whether there are pixels without color information in the frame to be presented; whether there are pixels with a preset transparency in the frame to be presented. If any of these determinations result in "yes", then the operation of acquiring at least one historical frame that is temporally adjacent to the frame to be presented is performed. That is, if any one of these determinations results in "yes", the operation of acquiring at least one historical frame that is temporally adjacent to the frame to be presented will be performed.

[0095] If the pixels of the image to be presented contain at least one of accurate motion information, depth information, and color information, or if at least one of accurate motion information, depth information, and color information can be directly determined based on the image to be presented, then flicker analysis can be performed based on this information.

[0096] However, during the rendering of virtual scenes, some optimization settings in the rendering engine, such as reducing the level of detail, may cause motion information of certain pixels in the image to be rendered to be ignored in order to improve performance. This results in pixels in the image that lack motion information. Consequently, it is impossible to determine whether a pixel in the image is flickering due to irregular movement based on motion information.

[0097] When the rendering engine has a low-precision depth buffer, it may fail to record the depth information of all pixels. Consequently, the depth information of some pixels in the rendered image may be ignored, resulting in pixels lacking depth information. Therefore, it becomes impossible to determine whether a pixel in the rendered image exhibits flickering due to changes in depth information.

[0098] Similarly, during the rendering process of a virtual scene, due to insufficient or inaccurate lighting settings in the virtual scene, there may be pixels in the image to be rendered that do not have color information. Therefore, in the image to be rendered, it is impossible to determine whether the pixel has flickering due to changes in color information.

[0099] For objects with high transparency in a virtual scene, the rendering engine typically involves complex motion information calculations, distance-to-camera sorting calculations, and color calculations to ensure accurate rendering and blending of these objects with other objects. If pixels of a highly transparent object cannot blend correctly with other objects due to rendering precision limitations, the motion, depth, and color information of the corresponding pixels in the rendered image may be incorrect or missing. Consequently, it becomes impossible to determine whether a pixel exhibits flickering based on motion, depth, or color information in the rendered image.

[0100] Based on this, it is determined whether it is necessary to obtain at least one historical frame that is sequentially adjacent to the image to be presented, so as to obtain more accurate motion information, depth information or color information of the pixels in the image to be presented based on the historical frame, and to provide a more accurate basis for pixel flicker analysis of the image to be presented.

[0101] Because anomalies may occur in the aforementioned motion, depth, and color information, in one feasible approach, for pixels in the image to be presented that possess motion, depth, or color information, the existing motion, depth, or color information can be verified based on the results obtained from the pixel flicker analysis to ensure the accuracy of the motion, depth, or color information of the pixels in the image to be presented. Specifically, the existing motion, depth, or color information is compared with the results obtained from the pixel flicker analysis. For example, the existing motion information of a pixel in the image to be presented is compared with the motion information of that pixel obtained from the motion analysis of the image. If the comparison result exceeds a preset value, the existing motion information of that pixel can be considered unreliable, and the existing motion information of that pixel is replaced with the motion information of that pixel obtained from the motion analysis of the image. In practical applications, the comparison of existing motion, depth, or color information with the results obtained from the pixel flicker analysis can be implemented by those skilled in the art in any appropriate manner according to actual needs, such as calculating the difference or quotient.

[0102] Step S204. Based on the results of pixel flicker analysis, determine the flickering pixels to be processed in the image.

[0103] In a virtual scene, some pixels may exhibit unexpected flickering, including but not limited to: flickering pixels that do not flicker according to the expected pattern, and flickering pixels that are not expected to flicker but actually flicker in the virtual scene. Pixels exhibiting unexpected flickering need to be identified as flickering pixels to be processed in subsequent steps, thereby improving the presentation quality of the image to be displayed. For example, for virtual shooting purposes, a glass curtain wall exists in the virtual scene, and the color of the glass curtain wall alternates between red and blue at a fixed frequency. In the virtual scene, the pixels constituting the glass curtain wall are expected to change color between red and blue at the same frequency, thus producing pixel flickering. However, due to the unstable position of the light source, the color information of the pixels constituting the glass curtain wall changes irregularly, inconsistent with the expected color information change pattern, i.e., unexpected flickering occurs, affecting the quality of video frames obtained during virtual shooting based on the image to be displayed. Therefore, the pixels that produce unexpected flickering phenomena among the pixels constituting the glass curtain wall can be considered as flickering pixels to be processed in subsequent steps to improve the presentation quality of the image to be presented.

[0104] In one example, corresponding to the aforementioned motion analysis, depth analysis, and color analysis, the result of pixel flicker analysis may include at least one of the following: the result of motion analysis, the result of depth analysis, and the result of color analysis.

[0105] Pixel flicker analysis can reveal the dynamic changes in at least one aspect of motion, depth, and color of pixels in a virtual scene. For example, if the results of pixel flicker analysis include motion analysis, meaning they include motion information, then the results can show the dynamic changes in motion information of pixels in the virtual scene over the entire time sequence. Based on this, a judgment is made on these dynamic change characteristics. If, during the process of displaying a virtual scene from a previous frame to the frame to be displayed, at least one of the motion, depth, or color information of pixels in the frame to be displayed changes, and this change does not conform to the expected pattern when the virtual scene was created, then it can be considered a flickering pixel to be processed, requiring further processing in subsequent steps to improve the presentation quality of the frame to be displayed. For example, if the color information of pixel A was expected to remain unchanged when the virtual scene was created, but the pixel flicker analysis results show that the color information of pixel A changed in the 5th frame out of 10 consecutive frames, then based on the results of the pixel flicker analysis, the color information of pixel A did not follow the expected pattern, and therefore pixel A can be considered to have produced an unexpected flickering phenomenon, making pixel A a flickering pixel to be processed.

[0106] Therefore, this step uses the results of pixel flicker analysis to determine the flickering pixels to be processed in the image to be presented. Any one of the results of pixel flicker analysis can achieve the effect of determining the flickering pixels to be processed in the image to be presented, but using multiple results simultaneously can more comprehensively confirm the flickering pixels to be processed in the image to be presented.

[0107] In one feasible approach, the rendering trajectory of pixels in the image can be obtained based on the results of pixel flicker analysis; based on the rendering trajectory, flicker pixel detection is performed on the pixels in the image to obtain real flicker pixels and unexpected flicker pixels, and the unexpected flicker pixels are taken as flicker pixels to be processed.

[0108] In some scenarios, there may be real flickering pixels in the image, such as flashing warning lights or flashing light sources. The flickering of the pixels corresponding to these objects is regular and can be considered real flickering pixels. Conversely, they can be considered as unexpected flickering pixels, that is, abnormal flickering pixels that may be caused by different reasons and need to be flicker removed. These pixels can be regarded as flickering pixels to be processed.

[0109] The rendering trajectory of pixels can be considered as a visual representation of the dynamic changes of pixels in a virtual scene as the scene transitions from a previous view to the current view. It can be presented in any suitable form, such as a polygonal line or a smoothed curve, etc.

[0110] For example, when pixel flicker analysis results include motion analysis results, i.e., motion information, the rendering trajectory of pixels in the image can be a visual representation of the dynamic changes in pixel motion information. In one feasible approach, arranging the motion information of pixels in each frame in temporal order yields a series of motion information data points. Drawing and connecting these data points provides the rendering trajectory of the pixels in the image. For instance, if a sphere exists in a virtual scene and moves at a constant velocity 'a' in 11 consecutive frames, then the corresponding pixels also move at a constant velocity 'a'. Therefore, the velocity of the pixels corresponding to the sphere in each frame is 'a'. Drawing and connecting the velocities of the pixels corresponding to the sphere in each frame in temporal order yields the rendering trajectory in a two-dimensional coordinate system, as shown below. Figure 3A As shown in the figure, the x-axis represents the frame number, and the y-axis represents the pixel's movement speed in meters per second.

[0111] When pixel flicker analysis results include depth analysis results, i.e., when depth information is included, the rendering trajectory of pixels in the image can be a visual representation of the dynamic changes in pixel depth information. In one feasible approach, the depth information of pixels in each frame is arranged in temporal order to obtain a series of depth information data points. Drawing and connecting these data points yields the rendering trajectory of the pixels in the image. For example, in a virtual scene, there exists a sphere. In 11 consecutive frames, the sphere moves back and forth between a point far from and near the virtual camera. The depth value of the pixel corresponding to the sphere changes in each frame. Drawing and connecting the depth information of the sphere's corresponding pixels in each frame in temporal order yields the rendering trajectory in a two-dimensional coordinate system, as shown below. Figure 3B As shown in the figure, the x-axis represents the frame number, and the y-axis represents the pixel depth value, in millimeters.

[0112] When pixel flicker analysis results include color analysis results, i.e., color information, the rendering trajectory of pixels in the image can be a visual representation of the dynamic changes in pixel color information. In one feasible approach, arranging the color information of pixels in each frame in temporal order yields a series of color information data points. Drawing and connecting these data points provides the rendering trajectory of the pixels in the image. For example, in a virtual scene, a sphere exists. In 11 consecutive frames, the sphere's color switches between red and blue. The color information of the pixels corresponding to the sphere changes in each frame. Drawing and connecting the color information of the pixels corresponding to the sphere in each frame in temporal order yields the rendering trajectory in a two-dimensional coordinate system, as shown below. Figure 3C As shown in the figure, the x-axis represents the frame number, and the y-axis represents the weighted sum of the R, G, and B color values ​​of the pixel. In one feasible approach, the weights of the R, G, and B color values ​​are set according to the human eye's sensitivity to different colors. For example, a higher weight is assigned to colors that the human eye is more sensitive to, and a lower weight is assigned to colors that the human eye is less sensitive to. In this case, when the color information of the pixel changes, the weighted sum can better reflect the human eye's sensitivity to pixel flicker caused by changes in color information.

[0113] As mentioned earlier, in practice, flickering pixels include both genuine flickering pixels and unexpected flickering pixels. Genuine flickering pixels are generated based on the needs of virtual shooting. They flicker in the virtual scene following the expected pattern when the virtual scene was created, and therefore do not require processing. For example, in a virtual scene, there might be a sign that changes color every minute, switching between yellow and white. Correspondingly, the pixels corresponding to the sign also change color every minute, switching between yellow and white. In this case, the pixels corresponding to the sign flicker in the virtual scene. However, if flickering pixel detection is performed on the pixels corresponding to the sign based on the presentation trajectory, and the detection result shows that the pixels corresponding to the sign conform to the color change pattern, that is, they flicker according to the expected pattern when the virtual scene was created, then the pixels corresponding to the sign can be identified as genuine flickering pixels, which do not affect the presentation quality of the image to be presented, and therefore do not need to be removed. Unexpected flickering pixels, on the other hand, do not conform to their corresponding motion patterns. Therefore, unexpected flickering pixels are treated as flickering pixels to be processed and removed in subsequent steps. For example, in a virtual scene, there is a sphere that moves at a constant speed 'a' in 11 consecutive frames. It is expected that the pixels corresponding to the sphere will also move at a constant speed 'a'. However, according to the rendering trajectory, the pixels corresponding to the sphere are detected to be flickering. It is found that after the 5th frame, the speed of the pixels corresponding to the sphere is 2a. The dynamic change of the pixel motion information does not conform to its corresponding motion law. That is, it is flickering in violation of the expected law when the virtual scene was created. Therefore, the pixels corresponding to the sphere can be identified as unexpected flickering pixels, which affect the rendering quality of the image to be presented and need to be removed.

[0114] Based on the above principle, according to the presentation trajectory, flickering pixel detection is performed on the pixels in the image. Specifically, by judging whether the dynamic changes of the pixels in the image conform to the expected rules when establishing the virtual scene, the real flickering pixels and the unexpected flickering pixels are obtained. The unexpected flickering pixels are identified as flickering pixels to be processed in the subsequent processing, thereby improving the presentation quality of the image to be presented.

[0115] When pixel flicker analysis results include multiple factors such as motion analysis, depth analysis, and color analysis—that is, multiple types of motion, depth, and color information—the rendering trajectories of pixels at different levels in the image can be obtained. When detecting flickering pixels in the image, these different rendering trajectories can complement each other, thus more comprehensively identifying the flickering pixels to be processed in the image to be presented. For example, in 11 consecutive frames, there is a sphere. When creating a virtual scene, it is expected that the sphere's depth and motion information remain unchanged. When performing flickering pixel detection based on the rendering trajectory obtained from the depth information, the depth information of the sphere's pixels conforms to the expected pattern. However, when performing flickering pixel detection based on the rendering trajectory obtained from the motion information, it is confirmed that the corresponding pixels of the sphere have changed their motion information in the 5th frame, which does not conform to the expected pattern. Therefore, the rendering trajectory obtained from the motion information, as a supplement to the rendering trajectory obtained from the depth information, confirms that the sphere's pixels are unexpected flickering pixels, thus more comprehensively identifying the flickering pixels to be processed in the image to be presented.

[0116] In another feasible approach, the flickering pixel region to be processed can be determined based on the results of pixel flickering analysis; and the pixels in the flickering pixel region to be processed can be identified as the flickering pixels to be processed.

[0117] In virtual scenes, there may be numerous details and elements, making pixel flickering retrieval across the entire screen computationally expensive. Furthermore, certain areas require focused attention, where pixel flickering is more likely; for example, jagged edges in different areas can cause flickering. Therefore, through human-computer interaction, the flickering pixel areas can be identified within the displayed image. Pixels within these areas are then designated as flickering pixels, allowing for more flexible processing in subsequent steps, thus improving efficiency and reducing post-processing workload.

[0118] Step S206. Perform flicker removal processing on the flickering pixels to be processed, and display the image after flicker removal processing on the screen.

[0119] For flicker removal of flickering pixels, a uniform method or algorithm can be used. However, this is not the only approach. To achieve more precise flicker removal for different flickering pixels, one feasible method is to apply flicker removal intensities to the pixels according to different intensities. The flicker removal intensities for different flickering pixels can be determined based on statistical results of flicker removal intensities of similar flickering pixels, historical experience, or simulation results.

[0120] Optionally, in one feasible approach, a corresponding flicker removal intensity can be determined from multiple different flicker removal intensities based on the presentation trajectory of the flickering pixel to be processed; and flicker removal processing can be performed on the flickering pixel to be processed according to the determined flicker removal intensity.

[0121] In the image to be presented, the flickering pixels may exhibit varying degrees of flickering. This is reflected in the presentation trajectory as the degree to which the motion, depth, or color information of the flickering pixel at a given point deviates from the motion, depth, or color information that should be presented according to its corresponding change pattern. When performing flicker removal processing on the flickering pixels, pixels with weaker flickering should be processed with a weaker flicker removal intensity, and pixels with stronger flickering should be processed with a stronger flicker removal intensity. If a uniform removal intensity is applied, pixels with weaker flickering may experience blurring or distortion due to excessively strong removal intensity, while pixels with stronger flickering may fail to effectively remove the flickering due to insufficient removal intensity. The determination of flicker intensity can be flexibly set by those skilled in the art according to actual needs. For example, multiple deviation ranges can be set from low to high. If the flickering falls within a lower deviation range, it is considered weak; if it falls within a higher deviation range, it is considered strong; and if it falls within a range between the two, it is considered moderate. Similar to the foregoing, the specific settings for the deviation range and intensity level can be implemented by those skilled in the art according to actual needs, and the embodiments of this application do not impose any restrictions on this.

[0122] Based on this, in one feasible approach, multiple different flicker removal intensities are preset for different flicker levels to optimize the flicker removal effect on the image to be presented. For example, three flicker removal intensities are preset. Taking the motion information of pixels in the image to be presented as an example, when the motion information of the flickering pixel to be processed deviates from the motion information it should present according to the corresponding change rule by less than or equal to 5%, that is, compared to the motion information of the previous frame in time sequence, the increase or decrease in the motion information of the flickering pixel to be processed is less than or equal to 5%, the flickering degree of the flickering pixel to be processed is slight, and the corresponding flicker removal intensity is level three. At this time, the flickering pixel to be processed does not affect the user's viewing experience, and image details should be preserved as much as possible. When the motion information of the flickering pixel to be processed deviates from the motion information it should present according to the corresponding change rule by more than 5% and less than or equal to 20%, that is, compared to the motion information of the previous frame in time sequence... When the increase or decrease in motion information of the flickering pixel to be processed is greater than 5% and less than or equal to 20%, the flickering degree of the flickering pixel to be processed is obvious, and the corresponding flicker removal intensity is level two. At this time, the flickering pixel to be processed may attract the user's attention, and appropriate flicker removal should be performed, while preserving the details of the image as much as possible. When the degree of deviation of the motion information of the flickering pixel to be processed from the motion information that should be presented according to the corresponding change law is greater than 20%, that is, when the increase or decrease in motion information of the flickering pixel to be processed is greater than 20% compared with the motion information of the previous frame in the time sequence, the flickering degree of the flickering pixel to be processed is severe, and the corresponding flicker removal intensity is level one. At this time, the flickering pixel to be processed seriously affects the user's viewing experience and needs to be completely removed.

[0123] Therefore, based on the rendering trajectory of the flickering pixel to be processed, the corresponding flicker removal intensity can be determined from multiple different flicker removal intensities, thereby removing the flickering phenomenon more effectively.

[0124] In one feasible approach, a flicker removal strategy is selected based on a determined flicker removal intensity to remove flicker from the pixel to be processed. For example, if the determined flicker removal intensity is level three, and the flickering degree of the pixel to be processed is slight, the image to be presented can be smoothed. This is achieved by calculating the average value of pixels within a preset range around the pixel to be processed and replacing the value of the pixel to be processed. Alternatively, image enhancement techniques, such as increasing the contrast of the image, can be used to preserve as much detail as possible in the image to be presented while removing flicker. If the determined flicker removal intensity is level two, and the flickering degree of the pixel to be processed is significant, adaptive filtering can be applied to the image to be presented. Specifically, the filter parameters are dynamically adjusted based on the flickering degree and texture features in the image to be presented, thereby maintaining good image quality while removing flicker. If the determined flicker removal intensity is level one, and the flickering degree of the corresponding flickering pixel to be processed is severe, the flickering pattern of the flickering pixel to be processed can be learned by image processing algorithms such as Convolutional Neural Networks (CNN) and Generative Adversarial Networks (GAN), or a model can be trained by deep learning technology to identify and remove the flickering pixel to be processed, thereby effectively removing the flickering pixel to be processed.

[0125] After removing flickering pixels from the displayed image using the method described above, the resulting image is presented on the screen. At this point, there are no longer any flickering pixels, and the image quality is relatively high, the image is more stable, and the overall presentation is better. Based on this, the video obtained through virtual shooting using this displayed image is also of higher quality.

[0126] As can be seen from the above, the solution of this application embodiment, in the method of presenting a virtual scene through a screen, or projecting a virtual scene as a shooting background using a screen as an intermediary to achieve virtual shooting, performs pixel flicker analysis on the image to be presented on the screen used to present the virtual scene. Based on the result of the pixel flicker analysis, the flickering pixels in the image to be processed are identified, and then flicker removal processing is performed. Thus, the flickering pixels are processed before the image is presented on the screen, resulting in a flicker-removed image when the image is presented on the screen. That is, there are no flickering pixels in the image, but rather the image quality is relatively high, the image presentation is relatively stable, and the presentation effect is better. Based on this, the quality of the video obtained by virtual shooting based on the image presented on the screen is also higher.

[0127] Compared to traditional methods that use hardware to remove flicker, the solution in this application does not require additional hardware or modifications to the hardware. This not only improves the quality of the presented image but also reduces the implementation cost of flicker removal. Compared to traditional methods that use complex rendering algorithms to remove flicker, the solution in this application is simpler, more efficient, and more versatile.

[0128] Reference Figure 4 This illustration shows a structural schematic diagram of an electronic device according to an embodiment of this application. The specific embodiments of this application do not limit the specific implementation of the electronic device. Exemplarily, the electronic device can be implemented as follows: Figure 1A , or, such as Figure 1B The screen shown.

[0129] like Figure 4 As shown, the electronic device may include: a processor 402, a communications interface 404, a memory 406, and a communications bus 408.

[0130] in:

[0131] The processor 402, communication interface 404, and memory 406 communicate with each other via communication bus 408.

[0132] Communication interface 404 is used to communicate with other electronic devices or servers.

[0133] The processor 402 is used to execute program 410, specifically to perform the relevant steps in any of the above method embodiments.

[0134] Specifically, program 410 may include program code that includes computer operation instructions.

[0135] Processor 402 may be a CPU, a GPU (Graphics Processing Unit), an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application. The one or more processors included in the smart device may be processors of the same type, such as one or more CPUs; or they may be processors of different types, such as one or more CPUs and one or more ASICs.

[0136] Memory 406 is used to store program 410. Memory 406 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0137] Program 410 may include multiple computer instructions. Specifically, program 410 may use multiple computer instructions to cause processor 402 to perform the operation corresponding to any of the methods described in the foregoing multiple method embodiments.

[0138] The specific implementation of each step in procedure 410 can be found in the corresponding descriptions of the steps and units in the above method embodiments, and has corresponding beneficial effects, which will not be repeated here. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding process descriptions in the foregoing method embodiments, and will not be repeated here.

[0139] This application also provides a computer storage medium storing a computer program thereon, which, when executed by a processor, implements the method described in any of the foregoing method embodiments. The computer storage medium includes, but is not limited to, compact disc read-only memory (CD-ROM), random access memory (RAM), floppy disk, hard disk, or magneto-optical disk.

[0140] This application also provides a computer program product, including computer instructions that instruct a computing device to perform an operation corresponding to any of the methods in the above-described multiple method embodiments.

[0141] Furthermore, it should be noted that the user-related information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to image data for collection, data for analysis, data for storage, data for display, etc.) involved in the embodiments of this application are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0142] It should be noted that, depending on the implementation needs, the various components / steps described in the embodiments of this application can be broken down into more components / steps, or two or more components / steps or parts of the operation of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of this application.

[0143] The methods described in the embodiments of this application can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code downloaded over a network that is originally stored in a remote recording medium or a non-transitory machine-readable medium and will be stored in a local recording medium. Thus, the methods described herein can be stored on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an Application Specific Integrated Circuit (ASIC) or a Field Programmable Gate Array (FPGA)). It is understood that the computer, processor, microprocessor controller, or programmable hardware includes storage components (e.g., Random Access Memory (RAM), Read-Only Memory (ROM), Flash Memory, etc.) capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods described herein. Furthermore, when a general-purpose computer accesses code used to implement the methods shown herein, the execution of the code transforms the general-purpose computer into a dedicated computer for executing the methods shown herein.

[0144] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for specific applications, but such implementations should not be considered beyond the scope of the embodiments of this application.

[0145] The above embodiments are only used to illustrate the embodiments of this application, and are not intended to limit the embodiments of this application. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the embodiments of this application. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of this application, and the patent protection scope of the embodiments of this application should be defined by the claims.

Claims

1. A picture processing method, comprising: performing pixel flicker analysis on a picture to be presented on a screen for presenting a virtual scene in a virtual shooting process, wherein the picture is a picture of a pre-constructed virtual scene under a shooting angle of a virtual camera; determining to-be-processed flicker pixels in the picture based on a result of the pixel flicker analysis; performing flicker removal processing on the to-be-processed flicker pixels, and presenting the picture after the flicker removal processing on the screen; wherein the flicker removal processing on the to-be-processed flicker pixels comprises: presetting a plurality of different flicker removal intensities for different flicker degrees, and performing flicker removal processing on the to-be-processed flicker pixels according to the different flicker removal intensities.

2. The method of claim 1, wherein, The pixel flicker analysis on the picture comprises at least one of motion analysis on the picture, depth analysis on the picture, and color analysis on the picture.

3. The method of claim 2, wherein, When the pixel flicker analysis comprises the motion analysis, the performing of the pixel flicker analysis on the picture to be presented on the screen for presenting the virtual scene comprises: obtaining at least one historical picture adjacent in time sequence to the picture to be presented; determining pixels corresponding to pixels in the picture to be presented in the at least one historical picture, and motion information of the corresponding pixels; determining motion information of the pixels in the picture to be presented based on the motion information of the corresponding pixels.

4. The method of claim 3, wherein, The determining of the motion information of the pixels in the picture to be presented based on the motion information of the corresponding pixels comprises: performing weighted processing on the motion information of the pixels corresponding to the pixels in the picture to be presented in the at least one historical picture according to a time sequence relationship with the picture to be presented and a first preset weight; determining the motion information of the pixels in the picture to be presented according to a result of the weighted processing.

5. The method of claim 2, wherein, When the pixel flicker analysis comprises the depth analysis, the performing of the pixel flicker analysis on the picture to be presented on the screen for presenting the virtual scene comprises: obtaining at least one historical picture adjacent in time sequence to the picture to be presented; determining depth information of pixels corresponding to pixels in the picture to be presented in the at least one historical picture; determining depth information of the pixels in the picture to be presented based on the depth information of the corresponding pixels.

6. The method of claim 5, wherein, The determining of the depth information of the pixels in the picture to be presented based on the depth information of the corresponding pixels comprises: performing weighted processing on the depth information of the pixels corresponding to the pixels in the picture to be presented in the at least one historical picture according to a time sequence relationship with the picture to be presented and a second preset weight; determining the depth information of the pixels in the picture to be presented according to a result of the weighted processing.

7. The method of claim 2, wherein, When the pixel flicker analysis comprises the color analysis, the performing of the pixel flicker analysis on the picture to be presented on the screen for presenting the virtual scene comprises: obtaining at least one historical picture adjacent in time sequence to the picture to be presented; determine color information of a pixel corresponding to the pixel in the to-be-presented picture in the at least one historical picture; determine color information of the pixel in the to-be-presented picture based on the color information of the corresponding pixel.

8. The method of claim 7, wherein, The determining of the color information of the pixel in the to-be-presented picture based on the color information of the corresponding pixel includes: performing weighted processing on the color information of the pixel corresponding to the pixel in the to-be-presented picture in the at least one historical picture according to a time sequence relationship with the to-be-presented picture and a third preset weight; determine the color information of the pixel in the to-be-presented picture according to a result of the weighted processing.

9. The method of any of claims 3-8, wherein, Before the obtaining of the at least one historical picture adjacent in time sequence to the to-be-presented picture, the method further includes: determining at least one of the following: whether there is a pixel without motion information in the to-be-presented picture; whether there is a pixel without depth information in the to-be-presented picture; whether there is a pixel without color information in the to-be-presented picture; whether there is a pixel with a preset transparency in the to-be-presented picture; if there is a determination result of yes, performing the operation of obtaining the at least one historical picture adjacent in time sequence to the to-be-presented picture.

10. The method of any one of claims 3-8, wherein, The determining of the pixel corresponding to the pixel in the to-be-presented picture in the at least one historical picture includes: determining a position of the pixel in the to-be-presented picture, and determining a preset size of a pixel region containing the pixel according to the position; determining the pixel corresponding to the pixel in the to-be-presented picture from a region corresponding to the pixel region in the at least one historical picture.

11. The method of claim 1, wherein, The determining of the to-be-processed flicker pixel in the picture based on the result of the pixel flicker analysis includes: obtaining a presentation track of the pixel in the picture based on the result of the pixel flicker analysis; performing flicker pixel detection on the pixel in the picture according to the presentation track to obtain a real flicker pixel and an unexpected flicker pixel, and taking the unexpected flicker pixel as the to-be-processed flicker pixel.

12. The method of claim 1, wherein, The determining of the to-be-processed flicker pixel in the picture based on the result of the pixel flicker analysis includes: determining a to-be-processed flicker pixel region according to the result of the pixel flicker analysis; determining a pixel in the to-be-processed flicker pixel region as the to-be-processed flicker pixel.

13. The method of claim 1, wherein, The performing of the flicker removal processing on the to-be-processed flicker pixel according to different flicker removal intensities includes: determining a corresponding flicker removal intensity from a plurality of different flicker removal intensities according to a presentation track of the to-be-processed flicker pixel; performing the flicker removal processing on the to-be-processed flicker pixel according to the determined flicker removal intensity.

14. An electronic device comprising: a processor, a memory, a communication interface, and a communication bus, the processor, the memory, and the communication interface complete communication with each other through the communication bus; the memory is used to store at least one executable instruction, and the executable instruction causes the processor to perform an operation corresponding to the method in any one of claims 1-13.

15. A computer storage medium having stored thereon a computer program which, when executed by a processor, implements the method of any one of claims 1-13.

16. A computer program product comprising computer instructions instructing a computing device to perform operations corresponding to the method of any one of claims 1-13.

Images