Image processing method and device, and storage medium

Abstract

The invention discloses an image processing method and device, and a storage medium, and the method comprises the steps: determining a second frame image based on a motion vector MV of a first frame image, the second frame image being a frame insertion image of the first frame image and a third frame image, and the third frame image being a previous frame image of the first frame image; a plurality of user interfaces UIs included in a first rendering channel are merged to obtain a merged UI, and the first rendering channel is a rendering channel for rendering the first frame image; and performing combined rendering on the combined UI and the second frame image to obtain a target image.

Description

Technical Field

[0001] This application relates to image processing technology, and more particularly to an image processing method, apparatus, and storage medium. Background Technology

[0002] Image frame interpolation is a classic problem in computer vision. It can be simply described as generating an image at the intermediate time given two images at different moments. A common application of image frame interpolation is video upscaling. By interpolating frames into a video, a video that is originally low in frame rate or choppy can be transformed into a smoother video.

[0003] In related technologies, deep learning methods using optical flow information or image pixel information are used to obtain interpolated images, but these methods do not produce good results in obtaining interpolated images. Summary of the Invention

[0004] This application provides an image processing method, device, and storage medium that can obtain accurate frame interpolation results.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] This application provides an image processing method, the method comprising:

[0007] The second frame image is determined based on the motion vector (MV) of the first frame image. The second frame image is an interpolated image of the first frame image and the third frame image. The third frame image is the frame image preceding the first frame image.

[0008] The multiple user interface UIs included in the first rendering path are merged to obtain a merged UI. The first rendering path is the rendering path for rendering the first frame image.

[0009] The merged UI and the second frame image are merged and rendered to obtain the target image.

[0010] This application provides an electronic device, the electronic device comprising:

[0011] The determination module is configured to determine the second frame image based on the motion vector MV of the first frame image. The second frame image is an interpolated image of the first frame image and the third frame image, and the third frame image is the frame image preceding the first frame image.

[0012] The merging module is configured to merge multiple user interface UIs included in the first rendering path to obtain a merged UI, wherein the first rendering path is the rendering path for rendering the first frame image.

[0013] The rendering module is configured to merge and render the merged UI and the second frame image to obtain the target image.

[0014] This application provides an electronic device, including a processor, wherein the processor is configured to:

[0015] The second frame image is determined based on the motion vector MV of the first frame image. The second frame image is an interpolated frame image of the first frame image and the third frame image. The third frame image is the frame image preceding the first frame image.

[0016] The multiple user interface UIs included in the first rendering path are merged to obtain a merged UI. The first rendering path is the rendering path for rendering the first frame image.

[0017] The merged UI and the second frame image are merged and rendered to obtain the target image.

[0018] This application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps in the above-described image processing method.

[0019] This application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the above-described image processing method.

[0020] The chip provided in this application embodiment is used to implement the above-described image processing method. The chip includes a processor for calling and running a computer program from a memory, so that a device equipped with the chip performs the above-described image processing method.

[0021] The image processing method, device, and storage medium provided in this application determine a second frame image based on the motion vector (MV) of a first frame image. The second frame image is an interpolated image of the first frame image and a third frame image, and the third frame image is the frame image preceding the first frame image. Multiple user interface (UI) elements included in a first rendering path are merged to obtain a merged UI. The first rendering path is the rendering path that renders the first frame image. The merged UI and the second frame image are merged and rendered to obtain a target image. The UI in the generated interpolated image, i.e., the target image, is completely consistent with the UI in the original image, effectively avoiding UI tearing and misalignment. Furthermore, the determination of the interpolated image content through MV makes the interpolation result more accurate and the rendering effect smoother. Attached Figure Description

[0022] Figure 1 This is an optional flowchart illustrating the image processing method provided in an embodiment of this application;

[0023] Figures 2A to 2C These are optional schematic diagrams of the image content and UI provided in the embodiments of this application;

[0024] Figure 3 This is an optional flowchart illustrating the image processing method provided in an embodiment of this application;

[0025] Figures 4A to 4C This is an optional schematic diagram of the MV and motion mask provided in the embodiments of this application;

[0026] Figure 5 This is an optional flowchart illustrating the image processing method provided in an embodiment of this application;

[0027] Figure 6 This is an optional flowchart illustrating the image processing method provided in an embodiment of this application;

[0028] Figure 7 This is an optional flowchart illustrating the image processing method provided in an embodiment of this application;

[0029] Figure 8 This is an optional flowchart illustrating the image processing method provided in an embodiment of this application;

[0030] Figure 9 This is an optional schematic structural diagram of the electronic device provided in the embodiments of this application;

[0031] Figure 10 This is an optional schematic structural diagram of the electronic device provided in the embodiments of this application;

[0032] Figure 11 This is an optional schematic structural diagram of the electronic device provided in the embodiments of this application;

[0033] Figure 12 This is an optional schematic structural diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0035] The embodiments of this application can provide an image processing method, apparatus, and storage medium. In practical applications, the image processing method can be implemented by an electronic device, and the various functional entities in the electronic device can be collaboratively implemented by the hardware resources of computer equipment (such as terminal equipment, network equipment, etc.), such as computing resources like processors and communication resources (such as those used to support various communication methods such as optical fiber and cellular).

[0036] Of course, the embodiments of this application are not limited to providing methods and hardware, and can also be implemented in various ways, such as providing a storage medium (stored with instructions for executing the image processing method provided in the embodiments of this application).

[0037] The following describes various embodiments of the image processing method, apparatus, and storage medium provided in the embodiments of this application.

[0038] The image processing method provided in this application embodiment can be as follows: Figure 1 As shown, the method may include:

[0039] S101. The electronic device determines a second frame image based on the motion vector MV of the first frame image. The second frame image is an interpolated image of the first frame image and the third frame image. The third frame image is the frame image preceding the first frame image.

[0040] The first frame image is any frame image in the multimedia content to be displayed, and the frame preceding the first frame image in the multimedia content is the third frame image. In the embodiments of this application, the third frame image can be understood as the previous frame image, and the first frame image can be understood as the next frame image.

[0041] In this embodiment, MV is used to describe the transformation from one two-dimensional image to another, typically representing the positional offset of the image between two frames. The MV of the first frame image is used to represent the positional offset of the first frame image relative to the third frame image.

[0042] In this embodiment of the application, the size of the MV of the first frame image is the same as the size of the first frame image, representing the offset of each pixel in the first frame image, i.e., the position offset.

[0043] The electronic device determines the interpolated frame, i.e. the second frame, between the next frame and the previous frame based on the positional offset of the next frame relative to the previous frame.

[0044] In this embodiment, the first and second frame images represent image content and do not include interfaces that receive user operations, such as click controls and swipe controls, i.e., interfaces that interact with the user. In this embodiment, the interface that interacts with the user can also be understood as a user interface (UI).

[0045] In one example, for Figure 2AThe image content in the display interface 200 shown is as follows: Figure 2B The UI shown is not included; the UI in interface 200 is as follows: Figure 2C As shown.

[0046] S102, the electronic device merges the multiple user interface UIs included in the first rendering path to obtain a merged UI, and the first rendering path is the rendering path for rendering the first frame image.

[0047] The electronic device determines the render pass (RP) of the first frame image, i.e., the first RP, based on the shader and render target (RT) label information of the first frame image. A render pass can be understood as a path through which multiple drawing commands draw different objects to the same render target. The first RP can be understood as the RP that renders the first frame image.

[0048] The electronic device identifies multiple UIs included in the first RP and merges the multiple UIs based on the color information and transparency of each UI to obtain a merged UI.

[0049] In this embodiment, the electronic device can establish a pixel-level reference RT (Reference RT), and set the color information of the reference RT to the initial color information and the transparency of the reference RT to the initial transparency. Here, the pixel-level RT can be understood as a device used to record the color information and transparency of a pixel. For a single pixel, the electronic device merges the color information and transparency of multiple UIs to obtain the merged color information and transparency, and sets the color information and transparency of the reference RT corresponding to that pixel to the merged color information and transparency, thereby storing the merged color information and transparency, i.e., the color information and transparency of that pixel in the merged UI.

[0050] In this embodiment of the application, before S101, the electronic device can acquire the first RP and separate the image content and UI included in the first RP to obtain the first frame image and multiple UIs.

[0051] S103. The electronic device merges and renders the merged UI and the second frame image to obtain the target image.

[0052] The electronic device obtains the target image based on the color information and transparency of the merged UI, as well as the color information of the second frame image.

[0053] In this embodiment, the color information of the target image can be determined at the pixel level. Understandably, for each pixel in the second frame image, the final color information of that pixel, i.e., the target color information, is obtained based on the pixel's color information in the second frame image, the pixel's color information in the merged UI, and its transparency. The electronic device renders each pixel based on its target color information to obtain the target image.

[0054] For a pixel, its color information and transparency in the merged UI are labeled as RT.rgb and RT.w respectively. The target color information of this pixel in the second frame image is labeled as clolor. Then the target color information final.color of this pixel can be expressed as formula (1):

[0055] finalcolor=RT.rgb+RT.w*color formula (1).

[0056] The image processing method provided in this application can be applied to scenarios where there are dynamic objects in the image content, such as game scenes and security monitoring scenes.

[0057] In this embodiment of the application, the second frame image is an interpolated frame between the first frame image and the third frame image, which are determined by using the first frame image and the third frame image as the subsequent frame and the previous frame, respectively. The second frame image can be understood as a generated frame inserted between the two original frames, the first frame image and the third frame image.

[0058] exist Figure 1 In the image processing method shown, the interpolated frame and the merged UI are merged and rendered to obtain the final displayed interpolated frame image, i.e., the target image. In this embodiment of the application, the first frame image and / or the third frame image can also be merged and rendered with the merged UI to obtain the final displayed front frame image and / or back frame image.

[0059] In this embodiment, a second frame image is determined based on the motion vector (MV) of the first frame image. The second frame image is an interpolated image of the first frame image and the third frame image, and the third frame image is the frame image preceding the first frame image. Multiple user interface (UI) elements included in the first rendering path are merged to obtain a merged UI. The first rendering path is the rendering path that renders the first frame image. The merged UI and the second frame image are merged and rendered to obtain a target image. The UI in the generated interpolated image, i.e., the target image, is completely consistent with the UI in the original image, effectively avoiding UI tearing and misalignment. Furthermore, the determination of the image content for interpolation through MV makes the interpolation result more accurate and the rendering effect smoother.

[0060] In some embodiments, S101 determines the second frame image based on the motion vector MV of the first frame image, such as... Figure 3 As shown, it includes:

[0061] S301. Based on the motion mask of the first frame image, the MV is corrected to obtain the target MV;

[0062] S302. Based on the target MV, determine the fourth frame image, wherein the offset of the fourth frame image relative to the first frame image is the offset amount corresponding to the target MV;

[0063] S303. Based on the depth information of each pixel in the fourth frame image, determine the acquisition rules for each pixel in the fourth frame image. The acquisition rules are used to indicate the acquisition of color information of the corresponding pixel from the first frame image or the third frame image.

[0064] S304. Based on the acquisition rules of each pixel in the fourth frame image, color information is acquired to obtain the second frame image.

[0065] In this embodiment, a moving mask is used to mark the positions of dynamic objects such as people, vehicles, and props in the image content. The size of the moving mask for the first frame image is the same as the size of the first frame image, indicating whether each pixel in the first frame image belongs to a dynamic object or a dynamic region. In this embodiment, the value of the indication information corresponding to each pixel in the moving mask can be a first value or a second value, wherein the first value is used to indicate that the pixel does not belong to a dynamic object, and the second value is used to indicate that the pixel belongs to a dynamic object.

[0066] The electronic device corrects the moving mask (MV) of the first frame image based on the moving mask of the first frame image to obtain the target MV. In this embodiment, the MV can characterize whether the main region of the dynamic object in the first frame image is offset, but it cannot accurately reflect whether the edge of the dynamic object is offset. Therefore, the reflected offset situation deviates from the actual offset situation. The electronic device corrects the MV of the first frame image by using the moving mask of the first frame image, and the obtained target MV can accurately reflect the actual offset of the first frame image relative to the third frame image.

[0067] After determining the target MV, the electronic device determines a fourth frame image based on the target MV. The offset of the fourth frame image relative to the first frame image is less than the offset corresponding to the target MV. The fourth frame image can be understood as a temporarily determined image whose offset relative to the third frame image is less than the offset of the first frame image relative to the third frame image; that is, a frame image located between the predicted third frame image and the second frame image.

[0068] The electronic device determines the acquisition rules for each pixel in the fourth frame image based on the depth information of each pixel. In this embodiment, the acquisition rules are pixel-level, meaning the electronic device determines the acquisition rules for each pixel.

[0069] For a single pixel, if the acquisition rule for that pixel is to acquire its color information from the first frame image, then the electronic device acquires the color information of that pixel from the first frame image based on the acquisition rule for that pixel, and obtains the color information of that pixel in the second frame image.

[0070] For a single pixel, if the acquisition rule for that pixel is to acquire its color information from the third frame image, then the electronic device acquires the color information of that pixel in the third frame image based on the acquisition rule for that pixel, and obtains the color information of that pixel in the second frame image.

[0071] In this embodiment of the application, the color information of each pixel in the fourth frame image is obtained from the first frame image or the third frame image based on the offset of the pixel relative to the first frame image or the third frame image. The offset of the pixel can be determined by the offset of the pixel in the fourth frame image relative to the first frame image or the third frame image.

[0072] In this embodiment, the electronic device acquires the color information of corresponding pixels from the first or third frame image based on pixel-level acquisition criteria. This ensures that the color information of a pixel in the interpolated frame is not affected by other pixels, improving the accuracy of the interpolated color information and enhancing the image quality of the interpolated image. Furthermore, the interpolation determined by the image processing method provided in this embodiment, compared to 2D image algorithms in related technologies, has lower power consumption, more accurate results, better performance, and lower hardware entry barriers.

[0073] In some embodiments, S301 modifies the MV based on the motion mask of the first frame image to obtain the target MV, including:

[0074] If the motion mask is the same size as the dynamic region represented by the MV, then the MV is taken as the target MV;

[0075] If the size of the motion mask and the dynamic region represented by the MV are different, then the deviation between the motion mask and the MV is determined, and the offset information of the deviation part of the MV is adjusted to obtain the target MV.

[0076] In this embodiment of the application, the size of the dynamic region represented by the motion mask is greater than or equal to the size of the dynamic region represented by the MV. Here, the dynamic region can be understood as the region where the dynamic object is located.

[0077] If the size of the dynamic region represented by the motion mask is the same as the size of the dynamic region represented by the MV, it means that the dynamic region represented by the MV is accurate, and the MV of the first frame image is determined as the target MV.

[0078] If the size of the dynamic region represented by the motion mask is larger than the size of the dynamic region represented by the motion mask (MV), it indicates that the dynamic region represented by the MV is inaccurate. In this case, the value of the MV is adjusted based on the motion mask so that the size of the dynamic region represented by the MV is the same as the size of the dynamic region represented by the motion mask. Specifically, the electronic device determines the deviation of the MV relative to the motion mask based on the dynamic regions represented by the motion mask and the dynamic regions represented by the MV, and adjusts the offset information of the deviation of the MV so that the size of the dynamic region represented by the target MV is the same as the size of the dynamic region represented by the motion mask.

[0079] In one example, the dynamic region represented by MV is as follows: Figure 4A or Figure 4B As shown in 400, the dynamic region represented by the Moving Mask is as follows: Figure 4B As shown in 410, Figure 4C This is a schematic diagram illustrating the correction of the MV proposed in this application embodiment. If the size of the dynamic region represented by MovingMask is larger than the size of the dynamic region represented by MV, then there will be a deviation portion 420. In this case, MV400 can be corrected based on the deviation portion 420 so that the size of the dynamic region represented by MV is the same as the size of the dynamic region represented by MovingMask. The resulting corrected dynamic region represented by MV is as follows: Figure 4C As shown in Figure 430.

[0080] In this embodiment, for each pixel in the MV, the offset information includes: a horizontal offset, a vertical offset, and indication information indicating whether it belongs to a dynamic object. For each pixel in the MV, information from three channels is included to indicate the offset information of the corresponding pixel: a first channel indicating the horizontal offset, a second channel indicating the vertical offset, and a third channel indicating dynamic object information. For pixels in the compensation portion, their offset information is set to the offset information of the pixel closest to that pixel that belongs to the dynamic region.

[0081] In this embodiment, the three channels of each pixel in MV can be understood as r channel, g channel and b channel. The r channel and g channel store horizontal and vertical offset information, and the b channel is used to indicate whether it belongs to a dynamic object.

[0082] In this embodiment, the motion mask is used to correct the MV so that the corrected target MV can accurately reflect the actual offset.

[0083] In some embodiments, S302 determines a fourth frame image based on the target MV, including: for each pixel in the target MV, performing the following processing to construct the fourth frame image: determining a target offset based on an offset coefficient and the offset of the pixel in the target MV; determining the position information of a target pixel corresponding to the pixel in the fourth frame image based on the position information of the pixel and the target offset; and storing the pixel information of the pixel as the pixel information of the target pixel in the fourth frame image.

[0084] Here, based on each pixel in the target MV, the position of the corresponding target pixel in the fourth frame image is determined, and the pixel information of the target pixel is determined, thereby forming the fourth frame image.

[0085] For a pixel in the target MV, the offset of the pixel is multiplied by the pixel coefficient to obtain the target offset corresponding to the pixel. Based on the position of the pixel and the target offset, the position information of the target pixel corresponding to the pixel in the fourth frame image is determined, and the information of the pixel is used as the pixel information of the target pixel.

[0086] Taking the pixel at coordinate position (x0, y0) as an example, the offset of this pixel in the target MV is MV_final, where MV_final includes MV_final.x on the X-axis and MV_final.y on the Y-axis. Then the target offset can be expressed as formula (2):

[0087] offset = MV_final / offset coefficient formula (2).

[0088] The offset includes offset.x on the X-axis and offset.y on the Y-axis, where offset.x is MV_final.x / offset coefficient and offset.y is MV_final.y / offset coefficient.

[0089] In this embodiment, the offset coefficient is greater than 1 and can be set according to actual needs. For example, the offset coefficient can be 1.5, 2, etc.

[0090] After determining offset.x and offset.y, the target pixel coordinates corresponding to the pixel at coordinate position (x0, y0) in the fourth frame image are (x0-offset.x, y0-offset.y).

[0091] In this embodiment of the application, the pixel information may include the mask of the pixel in the motion mask and the depth information of the pixel.

[0092] In some embodiments, S302, based on the target MV, determines the fourth frame image, further including:

[0093] If multiple pixels in the target MV have the same position information, then the selected pixel is determined from the multiple pixels based on the pixel information of each pixel in the multiple pixels.

[0094] The pixel information of the selected pixel is stored as the pixel information of the target pixel in the fourth frame image.

[0095] If multiple pixels in the target MV have the same position information corresponding to the target pixel in the fourth frame, then select one pixel from the multiple pixels as the selected pixel, and use the pixel information of the selected pixel as the pixel information of the target pixel in the fourth frame image.

[0096] In one example, if the position information of the target pixel corresponding to the first pixel and the second pixel in the target MV is the same, then a candidate pixel is determined from the first pixel and the second pixel based on the pixel information of the first pixel and the pixel information of the second pixel; and the pixel information of the candidate pixel is stored as the pixel information of the target pixel in the fourth frame image.

[0097] In this embodiment of the application, the selection of the selected pixel is based on the pixel information of each pixel among multiple pixels, wherein the pixel information includes m-ask and depth information.

[0098] In one example, the mask and depth information are weighted to obtain a weighted result. The weighted results of each pixel in the multiple pixels are compared to select the selected pixel from the multiple pixels.

[0099] In one example, the higher priority information is compared based on the priority of the mask and depth information. If the selected pixel is not selected based on the higher priority information, and multiple candidate pixels are determined based on the higher priority information, the selected pixel is selected from the multiple candidate pixels based on the lower priority information.

[0100] For example, if the mask has a higher priority than the depth information, the masks of each pixel are compared first. If only one of the multiple pixels has a mask indicating that the pixel belongs to the dynamic region, then that pixel is selected as the candidate pixel. If there are multiple candidate pixels whose masks indicate that they belong to the dynamic region, then the depth information of each candidate pixel is compared, and the pixel whose depth information indicates that it is closest is selected as the candidate pixel.

[0101] In this embodiment, when multiple pixels have the same offset result, a pixel is selected from the multiple pixels based on the pixel information to offset to the target pixel, thereby resolving the pixel offset conflict problem.

[0102] In some embodiments, S303 determines the acquisition rules for each pixel in the fourth frame image based on the depth information of each pixel in the fourth frame image, including:

[0103] For each target pixel in the fourth frame image, the following processing is performed: determining the original pixel corresponding to the target pixel in the first frame image; determining the reference pixel in the fourth frame image that is at the same position as the original pixel; and determining the acquisition rule of the target pixel based on the depth of the target pixel in the fourth frame image and the depth of the reference pixel in the fourth frame image.

[0104] For a target pixel, the original pixel corresponding to that target pixel in the first frame image is determined. Here, the target pixel is obtained by subtracting the target offset from the pixel position information in the MV, and the position information of the original pixel is determined by adding the target offset to the target pixel's position information. The original pixel and the target pixel can be understood as pixels of the same object at different positions in different frame images, and the object's position is different in different frame images, that is, the position coordinates of the original pixel and the target pixel are different. The position offset of the original pixel and the target pixel represents the relative offset of the object in the first frame image and the fourth frame image.

[0105] After determining the original pixels, the electronic device determines a reference image in the fourth frame that is in the same position as the original pixels.

[0106] In one example, the target pixel's coordinates are (x1, y1). The original pixel in the first frame of the image corresponding to the target pixel has coordinates (x1+offest.x, y1+offset.y) in the first frame of the image, and the reference pixel is the pixel in the fourth frame of the image with coordinates (x1+offest.x, y1+offset.y).

[0107] The electronic device determines the depth of the reference pixel in the fourth frame image, i.e., the original depth, and determines the depth of the target pixel in the fourth frame image, i.e., the target depth, and determines the acquisition rules corresponding to the target pixel based on the original depth and the target depth.

[0108] In one example, the electronic device can determine the depth difference between the target depth and the original depth, and compare the depth difference with a depth difference threshold. If the depth difference is less than the depth difference threshold, the determined acquisition rule is used to indicate that the color information of the corresponding pixel is acquired from the first frame image. If the depth difference is greater than the depth difference threshold, the determined acquisition rule is used to indicate that the color information of the corresponding pixel is acquired from the third frame image. If the depth difference is equal to the depth difference threshold, the acquisition rule can be determined to indicate that the color information of the corresponding pixel is acquired from either the first frame image or the third frame image.

[0109] In this embodiment, the target depth of the target pixel and the original depth of the reference pixel at the same position as the original pixel are used to indicate the depth change corresponding to the offset of the target pixel. If the depth change is small, the color information of the target pixel is collected from the subsequent frame image. If the depth change is large, the color information of the target pixel is collected from the previous frame image.

[0110] In some embodiments, the fourth frame image may contain blank pixels. Blank pixels can be understood as pixels that do not correspond to the target offset of the target MV in the fourth frame image. That is, blank pixels do not have corresponding pixels in the target MV. The color information of these blank pixels can be collected from the previous frame image or the next frame image based on the settings.

[0111] In some embodiments, S102 merges the multiple user interface UIs included in the first rendering path to obtain a merged UI, including:

[0112] The color information of the merged UI is determined based on the color information and transparency of each UI in the plurality of UIs;

[0113] The transparency of the merged UI is determined based on the transparency of each UI element among the multiple UI elements.

[0114] In this embodiment of the application, when merging multiple UIs, the color information and transparency of the multiple UIs can be merged separately. Here, the merging of UIs is at the pixel level. For a single pixel, the color information and transparency of multiple UIs at that pixel are merged separately to obtain the color information and transparency of the merged UI at that pixel.

[0115] For color information, the color information of each UI in multiple UIs is weighted by transparency to determine the color information of the merged UI. all .finalcolor can be represented by formula (3):

[0116] UI all .finalcolor = UI n .alpha*UI n .color+(1-UI n .alpha)*UI n-1 .alpha*UI n- 1. color + ... + (1 - UI) n .alpha)(1-UI n-1 .alpha)(1-UI n-2 .alpha)…(1-UI2.alpha)UI1.alpha*UI1.color

[0117] Formula (3);

[0118] Among them, UI n .color represents the color information of the nth UI element. n .alpha represents the transparency of the nth UI element. n- 1. `color` represents the color information of the (n-1)th UI element. n-1 .alpha represents the transparency of the (n-1)th UI element, and so on.

[0119] Here, UI all .finalcolor can be understood as RT.rgb in formula (1).

[0120] In some embodiments, determining the transparency of the merged UI based on the transparency of each UI among the plurality of UIs includes: sequentially performing the following processing on each UI among the plurality of UIs until the last UI: determining the transparency difference between the transparency of the UI and a set transparency; fusing the transparency difference with a reference transparency to obtain a new reference transparency, wherein the reference transparency is the fusion result of the transparency of the merged UIs. In this embodiment, the set transparency can be 1 or other values.

[0121] For transparency information, the difference between the transparency of each UI element and the set transparency value can be multiplied to determine the transparency of the merged UI. The transparency of the merged UI can be expressed as...

[0122] Here This can be understood as RT.w in formula (1).

[0123] In this embodiment of the application, the UI will be... all .finalcolor contains the RGB channel of RT, which will The color information is stored in the alpha channel of RT and then merged with the color information of the second frame image to obtain the final color information of the output display image.

[0124] Understandably, the color information in the embodiments of this application includes RGB information of the R channel, G channel and B channel.

[0125] The image processing method provided in this application embodiment will be further explained below, taking a game scene as an example.

[0126] Game frame generation technology based on 2D images, whether through AI convolution or optical flow calculation, converts the color information of the game images of the two consecutive frames into block inputs, detects the pixel position offset of the two consecutive frames through a certain step size on the image, and generates the image of the intermediate frame after processing.

[0127] Because 2D image algorithms can only capture the color information of game frames and lose the true position information in the original game frames, they are prone to the following defects:

[0128] 1. The generated frame offset error is relatively large, and it is easy to get too close to the previous or subsequent frame, resulting in poor frame interpolation effect.

[0129] 2. Because game UI is usually drawn separately and last, it covers part of the game screen, has a lot of overlapping colors, and the offset between two frames is completely inconsistent with the game screen. Therefore, the image method cannot handle game UI well, and the UI will be torn or distorted.

[0130] 3. Poor handling of in-game objects with clear, straight boundaries, easily resulting in misalignment or distortion.

[0131] 4. Color changes in the game that are not caused by object movement, such as changes in the object's own color or color changes caused by full-screen post-processing effects, are prone to failure or generation errors.

[0132] 5. Due to the limitation of the detection step size, when the lens moves at a large angle or the object's position changes rapidly, the correct results cannot be obtained.

[0133] This application embodiment obtains 3D scene information during the game rendering process by probing the rendering pipeline, including game scene images of previous and subsequent frames, MV, semi-transparent textures, UI rendering pass, depth, moving mask, etc.

[0134] First, the UI and game content to be rendered are determined in the UI render pass and separated. Then, errors in the game MV are corrected by moving mask. An intermediate frame offset map is constructed using the MV. The rules for subsequent color sampling are determined by depth and semi-transparent texture. Finally, the game scene images of the previous and next frames are sampled using the rules and the intermediate frame offset map to obtain the intermediate frame image. The intermediate frame image and the separated UI are merged and rendered to obtain the final intermediate frame image.

[0135] The rendering process in related technologies is as follows: Figure 5 As shown, the process includes: rendering game screen 501, rendering UI1 based on the rendered game screen 501, resulting in screen 502 including the game screen and UI1, rendering UI2 based on screen 502, resulting in screen 503 including the game screen and UI1 and UI2, continuing to render the next UI on screen 503, and so on, until the final screen 504.

[0136] S502, Render the first user interface on the game screen;

[0137] S503: Render the next user interface on the game screen until all user interfaces are rendered, and obtain the final display screen.

[0138] In this embodiment of the application, the rendering process is as follows: Figure 6 As shown, the process includes: separating the game screen (raw frame) and the user interface in the rendering pipeline to obtain game screen 601 (raw frame) and at least one UI, where at least one UI includes UI1, UI2, UI3, ..., UIn. Based on UI1602, at least one UI is merged to obtain merged UI603. Game screen 601 and merged UI603 are merged and rendered to obtain final screen 604. Interpolated game screen 605 and merged UI603 are merged and rendered to obtain interpolated final screen 606.

[0139] In this embodiment, the game screen does not include a UI, but only image content, and the final screen is an overlay or composite of the game screen and the UI.

[0140] In this embodiment of the application, the formula for synthesizing the transparency (alpha) of a single drawing call is formula (4):

[0141] finalColor=UI.alpha*UI.color+Game.color*(1-UI.alpha) formula (4);

[0142] Here, UI.alpha is the transparency of the UI, UI.color is the color of the UI, Game.color is the color of the game screen, and finalColor is the color of the rendered display screen.

[0143] In this embodiment of the application, the formula for combining the transparency (alpha) of the two drawing commands (drawcall) is formula (5):

[0144] finalColor=UI2.alpha*UI2.color+(UI1.alpha*UI1.color+Game.color*(1-UI1.alpha)*(1-UI2.alpha)

[0145] Formula (5);

[0146] Where UI1.alpha is the transparency of UI1, UI1.color is the color of UI1, UI2.alpha is the transparency of UI2, and UI2.color is the color of UI2.

[0147] Similarly, the formula for combining the alpha of n drawing commands (drawcall) is formula (6):

[0148]

[0149] Among them, UI all .finalcolor is the final color of all UI elements combined. i .alpha represents the transparency of the i-th UI element, where i ranges from 1 to n. This means multiplying the results of subtracting the transparency of IU1, IU2, and so on down to IUn from 1.

[0150] The theoretical basis for separating the UI: [This refers to separating the UI from the rest of the text.] all .finalcolor contains the RGB channel of RT, which will The color is stored in the RT's alpha channel and then recombined, i.e.: finalColor = RT.rgb + RT.w * Game.color. The resulting finalColor is completely consistent with the finalColor used when rendering the UI in related technologies. Here, Game.color corresponds to color in formula (1), which refers to the color information of the game screen (excluding UI, only including image content) in the game scene.

[0151] The UI separation rendering process is as follows: Figure 7 As shown, it includes:

[0152] S701, Identify the UI render pass to be rendered.

[0153] S702. Create a new RT and modify the target RT of the UI to the new RT.

[0154] The new RT can be labeled as RT0, and the initial value of RT0 is initialized to (0, 0, 0, 1,). These four elements are the values ​​of the R channel, G channel, B channel and A channel (transparency channel), respectively.

[0155] S703: Mix the UI's RGB colors into the new RT's RGB channels with the same transparency.

[0156] S704: Incorporate the UI's transparency into the new RT transparency channel.

[0157] The new RT's alpha channel is blended in with the formula finalAlpha = (1 - src.alpha) * dst.alpha, where src.alpha is the alpha of the blended UI and dst.alpha is the current alpha of RT0.

[0158] If there are still unmerged UI elements, continue executing S703 and S704 until all UI elements are drawn.

[0159] S705: The new RT is composited and rendered from the game screen to obtain the final display interface.

[0160] Here, the game footage can be either the original frame or the interpolated frame.

[0161] In this embodiment of the application, the process for determining the frame interpolation can be as follows: Figure 8 As shown, it includes:

[0162] S801. Using shader tag and RT tag information, identify the moving mask and MV of the next frame of the frame to be interpolated, i.e., the first frame image.

[0163] S802. Compare the dynamic object part in the MV with the moving mask, and mark the final correct MV_final, i.e., the target MV.

[0164] The MV includes three channels: the r channel, the g channel, and the b channel. The r and g channels store offset information (the r channel stores horizontal offset information, and the g channel stores vertical offset information), and the b channel stores dynamic object information.

[0165] The b channel of MV contains information about whether it is a dynamic object. However, the information about dynamic objects indicated by the b channel may deviate from the actual situation in some cases (it will always be smaller than the correct value).

[0166] The dynamic region of the object represented by the moving mask is either larger or the same size as the dynamic region of the object represented by the moving mask.

[0167] The dynamic range object (MV) and moving mask are compared at the pixel level. If the MV and moving mask represent the same dynamic range of the object, the MV does not need correction. If the MV and moving mask represent different dynamic ranges, the MV is padded to compensate for the deviation. The compensation method is to make the RGB channel value of the deviation part equal to the RGB channel value of the nearest z channel (dynamic object) with a value of 1.

[0168] S803. Determine the intermediate frame offset map based on the target MV.

[0169] For the target MV, determine the pixel-level offset.

[0170] For a pixel in the first frame image, i.e., the original pixel, taking the position of the pixel in the first frame image, i.e., the original pixel position, as (x0, y0), the offset of each original pixel position is determined by offset = MV_final / offset coefficient, and the data of each original pixel position is stored in the corresponding offset position (x0 offset.x, y0 offset.y), i.e., the position of the target pixel.

[0171] If multiple original pixels correspond to the same target pixel, the mask and depth are compared based on their priorities to determine which original pixel's data should be stored at that location.

[0172] In this embodiment of the application, the target pixels that store the data of the original pixels constitute the intermediate frame offset map, i.e., the fourth frame image. The data corresponding to each pixel in the fourth frame image includes offset information, mask, and depth.

[0173] S804. Based on the intermediate frame offset map, determine the pixel-level acquisition rules.

[0174] Here, the capture rules are used to indicate whether to capture the color of the previous frame or the color of the next frame.

[0175] Taking the target pixel position (x1, y1) of the fourth frame image as an example, the offset and depth1 stored in the intermediate frame offset map are sampled. The original depth0 is collected with (x1+offset.x, y1+offset.y) as the coordinates of the reference pixel. Depth1 and depth1 are compared. If depth1-depth0 < the given threshold, it is determined to be the color of the subsequent frame; otherwise, it is determined to be the color of the previous frame.

[0176] S805: Based on pixel-level acquisition rules and intermediate frame offset map, acquire the color of the previous or next frame to obtain the interpolated image.

[0177] S806. Combine the interpolated image and the merged UI to obtain an interpolated image including the UI.

[0178] In this embodiment, the interpolated image determined in S806 is inserted between actual rendered frames, thereby doubling the frame rate and improving game smoothness.

[0179] The image processing method provided in this application includes the following technical effects:

[0180] 1. Separate, use, and re-composite the UI to make the UI in the generated frame completely consistent with the UI in the native screen, effectively avoiding UI tearing and misalignment.

[0181] 2. The screen space position offset obtained through MV is far more accurate and efficient than the position offset calculated by general 2D image algorithms, making the frame interpolation results more precise and the rendering effect smoother.

[0182] 3. It can obtain more information about the game's 3D scene, and can more accurately judge situations where the color changes but the position remains the same, thus avoiding rendering errors in such scenes.

[0183] 4. The MV is not limited by the range of camera movement in the game, and will not produce rendering errors in scenes with large camera movement angles.

[0184] 5. No need for color block convolution, optical flow calculation or other processing. The algorithm is more efficient than 2D frame interpolation in most scenarios, which can make the game frame rate more stable and power consumption lower.

[0185] 6. No additional external hardware is required; only the GPU is needed for computation, resulting in a low hardware entry barrier.

[0186] It should be noted that the image processing method provided in this application embodiment can be combined with super-resolution fusion to construct a super-frame and super-image fusion system, achieving a high-quality and smooth gaming experience with low power consumption.

[0187] An electronic device according to an embodiment of this application, such as Figure 9 As shown, the electronic device 900 includes:

[0188] The determining module 901 is configured to determine the second frame image based on the motion vector MV of the first frame image, wherein the second frame image is an interpolated frame image of the first frame image and the third frame image, and the third frame image is the frame image preceding the first frame image;

[0189] The merging module 902 is configured to merge multiple user interface UIs included in the first rendering path to obtain a merged UI, wherein the first rendering path is the rendering path for rendering the first frame image.

[0190] The rendering module 903 is configured to merge and render the merged UI and the second frame image to obtain the target image.

[0191] In some embodiments, the determining module 901 is further configured to:

[0192] Based on the motion mask of the first frame image, the MV is corrected to obtain the target MV;

[0193] Based on the target MV, a fourth frame image is determined, wherein the offset of the fourth frame image relative to the first frame image is less than the offset corresponding to the target MV;

[0194] Based on the depth information of each pixel in the fourth frame image, the acquisition rules for each pixel in the fourth frame image are determined. The acquisition rules are used to indicate the acquisition of color information of the corresponding pixel from the first frame image or the third frame image.

[0195] The second frame image is obtained by collecting color information based on the acquisition rules of each pixel in the fourth frame image.

[0196] In some embodiments, the determining module 901 is further configured to:

[0197] If the motion mask is the same size as the dynamic region represented by the MV, then the MV is taken as the target MV;

[0198] If the size of the motion mask and the dynamic region represented by the MV are different, then the deviation between the motion mask and the MV is determined, and the offset information of the deviation part of the MV is adjusted to obtain the target MV.

[0199] In some embodiments, the determining module 901 is further configured to:

[0200] For each pixel in the target MV, the following processing is performed to construct the fourth frame image:

[0201] The target offset is determined based on the offset coefficient and the offset of the pixel in the target MV;

[0202] Based on the pixel's position information and the target offset, the position information of the target pixel corresponding to the pixel in the fourth frame image is determined;

[0203] The pixel information of the pixel is stored as the pixel information of the target pixel in the fourth frame image.

[0204] In some embodiments, the determining module 901 is further configured to:

[0205] If multiple pixels in the target MV have the same position information, then the selected pixel is determined from the multiple pixels based on the pixel information of each pixel in the multiple pixels.

[0206] The pixel information of the selected pixel is stored as the pixel information of the target pixel in the fourth frame image.

[0207] In some embodiments, the determining module 901 is further configured to:

[0208] For each target pixel in the fourth frame image, the following processing is performed:

[0209] Determine the original pixel corresponding to the target pixel in the first frame image;

[0210] In the fourth frame image, identify the reference pixel whose position is the same as the original pixel;

[0211] The acquisition rules for the target pixel are determined based on the depth of the target pixel in the fourth frame image and the depth of the reference pixel in the fourth frame image.

[0212] In some embodiments, the merging module 902 is further configured to:

[0213] The color information of the merged UI is determined based on the color information and transparency of each UI in the plurality of UIs;

[0214] The transparency of the merged UI is determined based on the transparency of each UI element among the multiple UI elements.

[0215] In some embodiments, the merging module 902 is further configured to:

[0216] For each of the multiple UIs, perform the following processing sequentially until the last UI:

[0217] Determine the difference between the transparency of the UI and the set transparency value;

[0218] The transparency difference is then fused with the reference transparency to obtain a new reference transparency, which is the fusion result of the merged UI transparency.

[0219] In practical applications, the aforementioned modules such as the determination module, merging module, and rendering module can be implemented by processors located on electronic devices. Specifically, they can be implemented by a combination of one or more of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), Microprocessor Unit (MPU), Application Processor (AP), Digital Signal Processor (DSP), and Field Programmable Gate Array (FPGA).

[0220] Those skilled in the art should understand that the description of the electronic device in the embodiments of this application can be understood with reference to the description of the image processing method in the embodiments of this application.

[0221] Figure 10 A schematic diagram of an optional implementation of an electronic device is provided as an embodiment of this application, such as... Figure 10As shown, this application provides an electronic device 1000, including an electronic chip 1001. The electronic chip can be implemented as the image processing method described in one or more embodiments above. The electronic chip can be implemented as a processor of an electronic device or as a microprocessor of a fast charging module.

[0222] This application provides an electronic device. Figure 11 Another optional implementation provided for the embodiments of this application is a schematic diagram of the structure of an electronic device, such as... Figure 11 As shown, this application embodiment provides an electronic device 1100, including:

[0223] The processor 1101 and the storage medium 1102 storing the executable instructions of the processor 1101, the storage medium 1102 performing operations via the communication bus 1103 dependent on the processor 1101, when the instructions are executed by the processor 1201, perform the image processing method executed in one or more of the above embodiments.

[0224] It should be noted that in practical applications, the various components in the terminal are coupled together via the communication bus 1103. It can be understood that the communication bus 1103 is used to achieve communication between these components. In addition to the data bus, the communication bus 1103 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 11 The general labeled all buses as communication bus 1103.

[0225] This application provides a computer storage medium for storing a computer program that causes a computer to perform the steps of the image processing method as described in one or more of the above embodiments.

[0226] This application provides a schematic structural diagram of an electronic device 1200. Figure 12 The illustrated electronic device 1200 includes a processor 1210. The processor 1210 is configured to:

[0227] The second frame image is determined based on the motion vector MV of the first frame image. The second frame image is an interpolated frame image of the first frame image and the third frame image. The third frame image is the frame image preceding the first frame image.

[0228] The multiple user interface UIs included in the first rendering path are merged to obtain a merged UI. The first rendering path is the rendering path for rendering the first frame image.

[0229] The merged UI and the second frame image are merged and rendered to obtain the target image.

[0230] In this embodiment of the application, the processor 1210 can call and run a computer program from the memory to implement the image processing method in this embodiment of the application.

[0231] Optionally, such as Figure 12 As shown, the electronic device 1200 may further include a memory 1220. The processor 1210 can retrieve and run computer programs from the memory 1220 to implement the image processing method described in this embodiment.

[0232] The memory 1220 can be a separate device independent of the processor 1210, or it can be integrated into the processor 1210.

[0233] Optionally, such as Figure 12 As shown, the electronic device 1200 may also include a transceiver 1230, which the processor 1210 can control to communicate with other devices, specifically, to receive signals sent by other devices. Here, the transceiver may include at least two antennas.

[0234] Understandably, a transceiver includes multiple physical paths for receiving or transmitting signals, and the physical components on one or more physical paths for transmitting signals constitute a transmitter. The MPR value of each physical path in the transmitter is independent.

[0235] Optionally, the electronic device 1200 can implement the corresponding processes implemented by the electronic device in the various methods of the embodiments of this application. For the sake of brevity, these will not be described in detail here. It should be understood that the processor in the embodiments of this application may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method embodiments can be completed by the integrated logic circuit in the hardware of the processor or by instructions in the form of software. The processor mentioned above may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0236] It is understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DR RAM). It should be noted that the memory used in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0237] It should be understood that the above-described memory is exemplary and not a limiting description. For example, the memory in the embodiments of this application may also be static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DR RAM), etc. That is to say, the memory in the embodiments of this application is intended to include, but is not limited to, these and any other suitable types of memory.

[0238] This application also provides a computer-readable storage medium for storing computer programs.

[0239] Optionally, the computer-readable storage medium can be applied to the electronic device in the embodiments of this application, and the computer program causes the computer to execute the corresponding processes implemented by the electronic device in the various methods of the embodiments of this application. For the sake of brevity, it will not be described in detail here.

[0240] This application also provides a computer program product, including computer program instructions.

[0241] Optionally, the computer program can be applied to the electronic device in the embodiments of this application. When the computer program is run on a computer, it causes the computer to execute the corresponding processes implemented by the electronic device in the various methods of the embodiments of this application. For the sake of brevity, it will not be described in detail here.

[0242] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0243] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0244] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0245] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0246] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0247] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0248] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image processing method, characterized in that, The method includes: The second frame image is determined based on the motion vector MV of the first frame image. The second frame image is an interpolated frame image of the first frame image and the third frame image. The third frame image is the frame image preceding the first frame image. The multiple user interface UIs included in the first rendering path are merged to obtain a merged UI. The first rendering path is the rendering path for rendering the first frame image. The merged UI and the second frame image are merged and rendered to obtain the target image.

2. The method according to claim 1, characterized in that, The determination of the second frame image based on the motion vector MV of the first frame image includes: Based on the motion mask of the first frame image, the MV is corrected to obtain the target MV; Based on the target MV, a fourth frame image is determined, wherein the offset of the fourth frame image relative to the first frame image is less than the offset corresponding to the target MV; Based on the depth information of each pixel in the fourth frame image, the acquisition rules for each pixel in the fourth frame image are determined. The acquisition rules are used to indicate the acquisition of color information of the corresponding pixel from the first frame image or the third frame image. The second frame image is obtained by collecting color information based on the acquisition rules of each pixel in the fourth frame image.

3. The method according to claim 2, characterized in that, The process of correcting the MV based on the motion mask of the first frame image to obtain the target MV includes: If the motion mask is the same size as the dynamic region represented by the MV, then the MV is taken as the target MV; If the size of the motion mask and the dynamic region represented by the MV are different, then the deviation between the motion mask and the MV is determined, and the offset information of the deviation part of the MV is adjusted to obtain the target MV.

4. The method according to claim 2, characterized in that, Based on the target MV, the fourth frame image is determined, including: For each pixel in the target MV, the following processing is performed to construct the fourth frame image: The target offset is determined based on the offset coefficient and the offset of the pixel in the target MV; Based on the pixel's position information and the target offset, the position information of the target pixel corresponding to the pixel in the fourth frame image is determined; The pixel information of the pixel is stored as the pixel information of the target pixel in the fourth frame image.

5. The method according to claim 4, characterized in that, The step of determining the fourth frame image based on the target MV also includes: If multiple pixels in the target MV have the same position information, then the selected pixel is determined from the multiple pixels based on the pixel information of each pixel in the multiple pixels. The pixel information of the selected pixel is stored as the pixel information of the target pixel in the fourth frame image.

6. The method according to claim 2, characterized in that, The step of determining the acquisition rules for each pixel in the fourth frame image based on the depth information of each pixel in the fourth frame image includes: For each target pixel in the fourth frame image, the following processing is performed: Determine the original pixel corresponding to the target pixel in the first frame image; In the fourth frame image, identify the reference pixel whose position is the same as the original pixel; The acquisition rules for the target pixel are determined based on the depth of the target pixel in the fourth frame image and the depth of the reference pixel in the fourth frame image.

7. The method according to claim 1, characterized in that, The process of merging multiple user interface UIs included in the first rendering path to obtain a merged UI includes: The color information of the merged UI is determined based on the color information and transparency of each UI in the plurality of UIs; The transparency of the merged UI is determined based on the transparency of each UI element among the multiple UI elements.

8. The method according to claim 7, characterized in that, Determining the transparency of the merged UI based on the transparency of each UI in the plurality of UIs includes: For each of the multiple UIs, perform the following processing sequentially until the last UI: Determine the difference between the transparency of the UI and the set transparency value; The transparency difference is then fused with the reference transparency to obtain a new reference transparency, which is the fusion result of the merged UI transparency.

9. An electronic device, characterized in that, The electronic device includes: The determination module is configured to determine the second frame image based on the motion vector MV of the first frame image. The second frame image is an interpolated image of the first frame image and the third frame image, and the third frame image is the frame image preceding the first frame image. The merging module is configured to merge multiple user interface UIs included in the first rendering path to obtain a merged UI, wherein the first rendering path is the rendering path for rendering the first frame image. The rendering module is configured to merge and render the merged UI and the second frame image to obtain the target image.

10. An electronic device, comprising a processor, characterized in that, The processor is configured to: The second frame image is determined based on the motion vector MV of the first frame image. The second frame image is an interpolated frame image of the first frame image and the third frame image. The third frame image is the frame image preceding the first frame image. The multiple user interface UIs included in the first rendering path are merged to obtain a merged UI. The first rendering path is the rendering path for rendering the first frame image. The merged UI and the second frame image are merged and rendered to obtain the target image.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the image processing method according to any one of claims 1 to 8.

12. A storage medium storing an executable program, characterized in that, When the executable program is executed by the processor, it implements the image processing method according to any one of claims 1 to 8.

13. A chip, comprising a processor, characterized in that, The processor is configured to perform the image processing method according to any one of claims 1 to 8.

Images