Image processing method and electronic device

Abstract

The present application discloses an image processing method and an electronic device, which relate to the field of terminal technology, including: the electronic device can identify and separate translucent objects and non-transparent objects in an image frame, and for the translucent objects, perform mixed prediction based on their brightness values ​​to obtain a first prediction result corresponding to the translucent objects, wherein after the electronic device separates the translucent objects of the first prediction result, it obtains the first prediction result based on the translucent objects, and the first prediction result is a correct prediction frame of the translucent object, which has high accuracy; the target prediction image frame is obtained by mixing the first prediction result corresponding to the translucent object and the second prediction result corresponding to the non-transparent object, and the target prediction image frame takes into account the display characteristics of the translucent objects and the non-transparent objects in the image frame, and the target prediction image frame has high accuracy, and the picture is smoother during the display of the target prediction image frame.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of terminal technology, and in particular to an image processing method and an electronic device. Background Art

[0002] As end-side games are constantly updated and iterated, the rendering process of the game engine of the end-side device becomes more complex and diverse, and the resource load is also increasing. Due to the limited battery life of the end-side device, in order to reduce unnecessary rendering overhead without significantly affecting the game quality, predictive frame technology has become a more mainstream research technology. Predictive frame technology refers to learning from historical real image frames to generate future predicted image frames.

[0003] In order to enhance the user's immersive gaming experience, a large number of semi-transparent objects (such as shadows, special effects, particle effects, etc.) are added to the game scene rendering. When generating the predicted image frames of the game scene based on the predicted frame technology, the correct predicted image frames may not be obtained for semi-transparent objects, resulting in obvious image problems such as shadow jitter, special effects dislocation, etc. that are perceived by users. Summary of the invention

[0004] The embodiments of the present application provide an image processing method and an electronic device, which can identify and separate translucent objects and non-transparent objects in image frames in a rendered scene, and based on the image parameters of the translucent objects in the two previous and next image frames, predictive mixing processing is performed on the translucent object areas in the two previous and next image frames to obtain prediction results corresponding to the translucent objects. The method for separating and processing the semi-transparent objects can obtain relatively accurate prediction results for the semi-transparent object areas, and obtain predicted image frames based on the prediction results of the semi-transparent object areas and the prediction results of the non-transparent areas. The predicted image frames are relatively accurate, and the predicted image frames can be played by interpolating between real image frames to improve the smoothness and accuracy of the playback screen.

[0005] In order to achieve the above-mentioned purpose, the embodiments of the present application adopt the following technical solutions.

[0006] In a first aspect, an image processing method is provided, the method comprising:

[0007] The electronic device obtains a first image frame; the first image frame is the Nth image frame in the time sequence in drawing and rendering the scene.

[0008] Wherein, N is a natural number greater than 1; the first image frame includes translucent objects and non-transparent objects; the transparency of the translucent objects is greater than 0 and less than 1, and the transparency of the non-transparent objects is 1.

[0009] The electronic device performs recognition processing on the first image frame to obtain the semi-transparent objects and non-transparent objects in the first image frame.

[0010] The electronic device acquires a second image frame, where the second image frame is a previous image frame of the first image frame in time sequence.

[0011] The electronic device performs mixed prediction processing on the semi-transparent objects in the second image frame and the semi-transparent objects in the first image frame to obtain a first prediction result; performs prediction processing on the non-transparent objects in the second image frame and the non-transparent objects in the first image frame to obtain a second prediction result; and mixes the first prediction result with the second prediction result to obtain a first target prediction image frame. The electronic device performs rendering synthesis based on the semi-transparent objects and non-transparent objects in the first image frame to obtain a real image frame corresponding to the first image frame, and displays the first target prediction image frame between the second image frame and the real image frame.

[0012] The first prediction result represents the predicted mixed semi-transparent object area; the second prediction result represents the predicted non-transparent object area.

[0013] In the present application, the electronic device can identify and separate the translucent objects and non-transparent objects in the image frame, and for the translucent objects, perform a mixed prediction based on their brightness values ​​to obtain a first prediction result corresponding to the translucent object, wherein after the electronic device separates the translucent objects of the first prediction result, it obtains the first prediction result based on the translucent objects, and the first prediction result is the correct prediction frame of the translucent object, which has high accuracy and avoids the problem that the traditional prediction frame technology cannot generate a correct prediction image frame due to the loss of transparency information of the translucent objects; at the same time, the target prediction image frame is obtained by mixing the first prediction result corresponding to the translucent object and the second prediction result corresponding to the non-transparent object, and the target prediction image frame takes into account the display characteristics of the translucent objects and the non-transparent objects in the image frame, and the target prediction image frame has high accuracy. In the process of displaying the target prediction image frame, the picture is smoother, and the technology of predicting image frames can effectively reduce the power consumption of the electronic device in displaying image frames.

[0014] In a possible implementation manner of the first aspect, a first image frame includes a plurality of objects; and the electronic device performs recognition processing on the first image frame to obtain the semi-transparent object and the non-transparent object in the first image frame, including:

[0015] The electronic device identifies the mixing state corresponding to each object in the first image frame; the mixing state is used to characterize whether the object has been mixed; an object with an enabled mixing state is determined as a semi-transparent object, and an object with a disabled mixing state is determined as a non-transparent object.

[0016] In the present application, the electronic device can determine whether the object has been mixed and drawn based on the mixing state of each object, thereby determining that the object that has been mixed and drawn is a semi-transparent object, and the object that has not been mixed and drawn is a non-transparent object. The mixing state can effectively and accurately characterize the properties of the object, so that the electronic device can accurately and quickly determine whether the object is a non-transparent object or a semi-transparent object.

[0017] In another possible implementation manner of the first aspect, the first image frame corresponds to a first cache queue, and the first cache queue is used to store the rendered first image frame; the method further includes:

[0018] The electronic device creates a second cache queue and stores the semi-transparent objects in the rendered first image frame in the second cache queue.

[0019] Exemplarily, the electronic device unbinds the correspondence between the semi-transparent objects in the first image frame and the first cache queue, and establishes the correspondence between the semi-transparent objects in the first image frame and the second cache queue; the second cache queue is used to store the semi-transparent objects in the rendered first image frame.

[0020] In the present application, the translucent object is untied from the original corresponding first cache queue, and a corresponding relationship between a new second cache queue and the translucent object is established. In this way, the second cache queue only includes the rendering results of the translucent object, which can improve data processing efficiency in subsequent image processing.

[0021] In another possible implementation manner of the first aspect, the method further includes:

[0022] The electronic device stores non-transparent objects in the rendered first image frame in a first cache queue.

[0023] Exemplarily, the electronic device establishes a correspondence between the non-transparent objects in the first image frame and the first cache queue; the first cache queue is used to store the non-transparent objects in the rendered first image frame.

[0024] In the present application, the non-transparent objects are still bound to the original first cache queue, and there is no need to create a new third cache queue for storing rendered non-transparent objects, thereby reducing resource redundancy.

[0025] In another possible implementation manner of the first aspect, the electronic device performs hybrid prediction processing on the semi-transparent object in the second image frame and the semi-transparent object in the first image frame to obtain a first prediction result, including:

[0026] The electronic device determines a mixing coefficient based on a brightness difference between a first pixel included in a semi-transparent object area in a first image frame and a second pixel included in a semi-transparent object area in a second image frame; pixels at the same position of the first image frame and the second image frame have a corresponding relationship; and the mixing coefficient is used to characterize the ratio of a weighting coefficient of the first pixel and a weighting coefficient of the second pixel.

[0027] The electronic device obtains the pixel value of the third pixel based on the mixing coefficient, the pixel value of the first pixel, and the pixel value of the second pixel, and obtains the first prediction result based on the pixel value of the third pixel. The semi-transparent body area represented by the first prediction result is the same size as the semi-transparent object area in the first image frame and the semi-transparent object area in the second image frame.

[0028] In the present application, the electronic device determines a mixing coefficient based on the brightness difference of pixels of a semi-transparent object in a first image frame and a second image frame. The mixing coefficient can be dynamically adjusted based on the brightness difference of each pixel, thereby weakening the blurriness of the mixing result and making the first prediction result obtained based on the calculated pixel value of the third pixel more accurate.

[0029] In another possible implementation manner of the first aspect, the electronic device determines the mixing coefficient based on a brightness difference between a first pixel included in a region where the semi-transparent object is located in the first image frame and a second pixel included in a region where the semi-transparent object is located in the second image frame, including:

[0030] The electronic device obtains the brightness difference between the i-th first pixel and the i-th second pixel. If the brightness difference is less than or equal to a preset difference, the mixing coefficient is a first mixing coefficient, and the weighting coefficient of the first pixel in the first mixing coefficient is the same as the weighting coefficient of the second pixel. If the brightness difference is greater than the preset difference, the mixing coefficient is a second mixing coefficient, and the weighting coefficient of the first pixel in the second mixing coefficient is greater than the weighting coefficient of the second pixel.

[0031] The position of the i-th first pixel in the semi-transparent object area in the first image frame is the same as the position of the i-th second pixel in the semi-transparent object area in the second image frame.

[0032] In the present application, the brightness difference between the pixels of the semi-transparent object in the first image frame and the second image frame is less than or equal to the preset difference, indicating that there is no obvious brightness change between the semi-transparent object in the first image frame and the second image frame. In this case, the pixel value of the third pixel is obtained with a more balanced mixing coefficient, and the pixel value is relatively accurate. The brightness difference between the pixels of the semi-transparent object in the first image frame and the second image frame is greater than the preset difference, indicating that there is an obvious brightness change between the semi-transparent object in the first image frame and the second image frame. In order to make the connection between the image frames smoother, the mixing coefficient increases the proportion of the subsequent image frames in the time sequence, that is, the weighting coefficient of the first pixel of the first image frame is increased, and the pixel value of the third pixel calculated is a relatively accurate transition value.

[0033] In another possible implementation manner of the first aspect, the electronic device acquires a pixel value of a third pixel based on the mixing coefficient, the pixel value of the first pixel, and the pixel value of the second pixel, including:

[0034] The electronic device performs weighted processing on the pixel value of the i-th first pixel and the pixel value of the i-th second pixel based on the mixing coefficient to obtain the pixel value of the i-th third pixel.

[0035] In the present application, the mixing coefficient can be dynamically adjusted based on the brightness difference of each pixel, which weakens the blurriness of the mixing result and makes the first prediction result obtained based on the calculated pixel value of the third pixel more accurate.

[0036] In another possible implementation manner of the first aspect, the method further includes:

[0037] If the pixel values ​​of the j-th first pixel and the j-th second pixel are both null, the electronic device determines that the pixel value of the j-th pixel in the first prediction result is a null value.

[0038] Among them, the position of the jth first pixel in the semi-transparent object area in the first image frame, the position of the jth second pixel in the semi-transparent object area in the second image frame, and the position of the jth pixel in the semi-transparent object area of ​​the first prediction result are the same.

[0039] In the present application, if the pixel values ​​of the j-th first pixel and the j-th second pixel are both empty, it means that the pixel at this position in the first image frame and the second image frame is empty, then accordingly, the value of the third pixel corresponding to the position output is also empty.

[0040] In another possible implementation manner of the first aspect, the method further includes:

[0041] The electronic device obtains the brightness value of the i-th first pixel according to the pixel value of the i-th first pixel and the mapping relationship; the pixel value includes the RGB value, and the mapping relationship includes the conversion relationship between the RGB value of the pixel and the brightness value;

[0042] as well as,

[0043] The electronic device obtains the brightness value of the i-th second pixel according to the pixel value of the i-th second pixel and the mapping relationship.

[0044] In the present application, the electronic device can calculate the brightness value of each pixel based on the conversion mapping relationship between the RGB value of each pixel and the RBG value and the brightness value, thereby determining the mixing coefficient based on the difference in the brightness values ​​of pixels at the same position in the first image frame and the second image frame. This solution is simple and does not require complex computing resources.

[0045] In another possible implementation manner of the first aspect, the electronic device performs mixed processing on the first prediction result and the second prediction result to obtain a first target predicted image frame, including:

[0046] The electronic device uses an Alpha blending rendering method to blend the first prediction result with the second prediction result according to the transparency of the semi-transparent object in the first prediction result, and obtains a mixed pixel value of each pixel after the blending process; the electronic device obtains a first target predicted image frame according to the mixed pixel value of each pixel.

[0047] In the present application, since translucent objects have their own transparency mixing coefficient (Alpha value), the electronic device adopts an Alpha mixing rendering method to mix and render the first prediction result and the second prediction result. The blur caused by the mixing operation will be further weakened, and the obtained first target prediction image frame will be more accurate.

[0048] In another possible implementation manner of the first aspect, the electronic device performs prediction processing on the non-transparent object in the second image frame and the non-transparent object in the first image frame to obtain a second prediction result, including:

[0049] The electronic device adopts the motion estimation and compensation MEMC technology to predict the non-transparent object in the second image frame and the non-transparent object in the first image frame to obtain a second prediction result.

[0050] In the present application, for non-transparent objects, the electronic device can use traditional prediction frame technology to obtain the second prediction result. The traditional prediction frame technology is relatively mature, and the prediction result is relatively accurate for the image frame of the non-transparent object.

[0051] In another possible implementation manner of the first aspect, after acquiring the semi-transparent object and the non-transparent object in the first image frame, the method further includes:

[0052] The electronic device determines a blending decision result based on the translucent object in the first image frame and preset conditions; if the translucent object in the first image frame meets the preset conditions, the blending decision result is to perform blending processing; if the translucent object in the first image frame does not meet the preset conditions, the blending decision result is not to perform blending processing.

[0053] Among them, the preset conditions include that the number of valid pixels in the area where the translucent object is located accounts for a proportion of the total number of pixels on the screen of the electronic device that is greater than a preset threshold, and the valid pixels are pixels whose brightness values ​​are greater than the brightness threshold.

[0054] In the present application, when the color of some translucent objects included in the scene rendering is very light (brightness is less than a preset brightness threshold) or the proportion of the translucent objects in the screen space is less than a preset threshold, and the translucent objects do not have a significant impact on the scene rendering picture, in order to further reduce the image processing power consumption of the electronic device, there is no need to perform separate processing of the translucent objects, and the real image frame can be directly used for MEMC prediction.

[0055] After acquiring the translucent object, the electronic device can make a hybrid decision according to preset conditions to obtain a hybrid decision result. In the subsequent execution of the image processing method, the corresponding steps can be executed based on the hybrid decision result, further avoiding the redundancy of computing resources and reducing the power consumption of the electronic device.

[0056] In another possible implementation manner of the first aspect, the electronic device performs hybrid prediction processing on the semi-transparent object in the second image frame and the semi-transparent object in the first image frame to obtain a first prediction result, including:

[0057] When the hybrid decision result is to perform hybrid processing, the electronic device performs hybrid prediction processing on the semi-transparent object in the second image frame and the semi-transparent object in the first image frame to obtain a first prediction result.

[0058] In the present application, when the mixing decision result is to perform mixing processing, the electronic device performs mixed prediction processing on the semi-transparent objects in the second image frame and the semi-transparent objects in the first image frame to obtain the first prediction result. Targeted mixing processing of the semi-transparent objects is performed when necessary, thereby improving the effectiveness of the image processing method.

[0059] In another possible implementation manner of the first aspect, the method further includes:

[0060] When the mixing decision result is not to perform mixing processing, the electronic device uses motion estimation and compensation MEMC technology based on the real image frame and the second image frame to obtain the second target predicted image frame; the electronic device displays the second target predicted image frame between the real image frame and the second image frame.

[0061] In the present application, when the mixing decision result is not to perform mixing processing, the electronic device uses a traditional prediction frame method to acquire the second target prediction image frame. There is no need to perform targeted mixing processing of translucent objects, which reduces the redundancy of computing resources and further reduces the power consumption of the electronic device.

[0062] In a second aspect, an electronic device is provided, which includes a memory, a display screen and one or more processors; the memory and the display screen are coupled to the processor; the memory stores computer program code, and the computer program code includes computer instructions, and when the computer instructions are executed by the processor, the electronic device executes a method as described in any one of the above-mentioned first aspects.

[0063] According to a third aspect, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium, and when the computer-readable storage medium is executed on an electronic device, the electronic device can execute any one of the methods according to the first aspect.

[0064] According to a fourth aspect, a computer program product comprising instructions is provided, which, when executed on an electronic device, enables the electronic device to execute any one of the methods according to the first aspect.

[0065] In a fifth aspect, an embodiment of the present application provides a chip, the chip including a processor, the processor being used to call a computer program in a memory to execute the method of the first aspect.

[0066] It can be understood that the beneficial effects that can be achieved by the electronic device described in the second aspect, the computer-readable storage medium described in the third aspect, the computer program product described in the fourth aspect, and the chip described in the fifth aspect provided above can refer to the beneficial effects in the first aspect and any possible design method thereof, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 A schematic diagram of shadow jitter in a game scene provided by an embodiment of the present application;

[0068] Figure 2 A schematic diagram of a special effect separation in a game scene provided in an embodiment of the present application;

[0069] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0070] Figure 4 A schematic diagram of a system architecture of an electronic device provided in an embodiment of the present application;

[0071] Figure 5 A flowchart of an image processing method provided in an embodiment of the present application;

[0072] Figure 6 A schematic diagram of a process for obtaining a first prediction result in an image processing method provided in an embodiment of the present application;

[0073] Figure 7 A schematic diagram of the processing process of a MEMC prediction technology is provided for an embodiment of the present application;

[0074] Figure 8 A flowchart of another image processing method provided in an embodiment of the present application;

[0075] Fig. 9 A flowchart of another image processing method provided in an embodiment of the present application;

[0076] Fig.10 A schematic diagram of the structure of another electronic device provided in an embodiment of the present application;

[0077] Fig.11 A schematic diagram of the structure of a chip system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0078] In the description of the embodiments of the present application, the terms used in the following embodiments are only for the purpose of describing specific embodiments, and are not intended to be used as limitations to the present application. As used in the specification and the appended claims of the present application, the singular expressions "a", "said", "above", "the" and "this" are intended to also include such expressions as "one or more", unless there is a clear contrary indication in the context. It should also be understood that in the following embodiments of the present application, "at least one", "one or more" refer to one or more (including two). The term "and / or" is used to describe the association relationship of associated objects, indicating that three relationships can exist; for example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are a kind of "or" relationship.

[0079] References to "one embodiment" or "some embodiments" etc. described in this specification mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Thus, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways. The term "connection" includes direct connection and indirect connection, unless otherwise specified. "First" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated.

[0080] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0081] With the continuous update and iteration of electronic devices, the rendering process of the game engine of electronic devices has become more complex and diverse, and the resource load has also increased. Due to the limited battery life of electronic devices, in order to reduce unnecessary rendering overhead without significantly affecting the game quality, predictive frame technology has become a more mainstream research technology.

[0082] Predictive frame technology refers to learning from real image frames and inserting a predicted frame image between two real frame images, so predictive frame technology can also be called interpolation technology. Exemplarily, predictive frame technology includes frame rate conversion (FRC) technology, motion estimation and compensation (MEMC) technology, etc.

[0083] A predicted frame image is a predicted image generated between two real frame images. Inserting a predicted frame image between two real frames can improve the frame rate of the displayed image, thereby reducing the sense of screen freeze and reducing the power consumption of game rendering. The generation process of the predicted frame image includes: calculating the motion vector between the two real frame images, and then shifting the pixels or pixel blocks of the real frame image based on the motion vector. Under normal circumstances, pixels or pixel blocks that are moved out of the image range are directly discarded, while pixels that are moved from outside the image into the image range need to be specially processed, for example, pixel filling.

[0084] The types of prediction frame technology include intra-frame prediction and inter-frame prediction. If the prediction process information source comes from the current frame, it is called intra-frame prediction; if the prediction information source comes from other frames, it is called inter-frame prediction. Prediction frame technology was initially used to process image frames in videos, and then applied to the processing of image frames in animations. For example, prediction frame technology can be applied to image frame processing in video playback scenes, image frame processing in game animation scenes, and so on.

[0085] For example, in the scenario where the image frames in the game animation scene are predicted based on the traditional prediction frame technology, in order to enhance the user's game immersion experience, a large number of semi-transparent object drawings (such as shadows, special effects, particle effects, etc.) will be added to the game animation scene. Due to the lack of transparency information of these semi-transparent objects, the traditional prediction frame technology cannot obtain the correct prediction image frame, resulting in the appearance of shadow jitter, special effects dislocation and other obvious screen problems perceived by users, which greatly reduces the user experience.

[0086] For example, reference Figure 1 . Figure 1 A schematic diagram of shadow jitter in a game animation scene that appears after predicting an image frame of the game animation scene based on traditional prediction frame technology is given. Figure 1 The semi-transparent objects in the game animation scene include the "shadow / shadow" 101 of the character, wherein: Figure 1 (a) is a normal display screen. After the prediction frame image is processed based on the traditional prediction frame technology, the character's "shadow / shadow" will shake and produce afterimages. Figure 1 (b) in the figure; at the same time, the user can intuitively feel that the "shadow / shadow" of the person in the picture shakes and produces afterimages, which gives the user a bad user experience.

[0087] For example, reference Figure 2 . Figure 2A schematic diagram of the dislocation of special effects in a game animation scene after predicting the predicted image frame of the game animation scene based on traditional prediction frame technology is given. Figure 2 The semi-transparent objects in the game animation scene include special effects 201 of characters, wherein: Figure 2 (a) is a normal display screen. When the character moves, the special effect 201 should theoretically move with the character, and the relative distance between the special effect 201 and the character remains unchanged. After the prediction frame image is processed based on the traditional prediction frame technology, the special effect 201 of the character will appear to be separated from the character when the character moves. Figure 2 In (b), the special effects of the characters appear in the wrong position on the screen; at the same time, the user can intuitively feel that the special effects of the characters in the screen are out of place, which causes a bad user experience.

[0088] When processing predicted image frames containing translucent objects, traditional prediction frame technology does not pay attention to the problem of missing transparency information of semi-transparent objects. Direct prediction processing of the entire image frame is bound to lead to prediction failure or inaccurate prediction of semi-transparent objects in the image frame.

[0089] The embodiment of the present application provides an image processing method, which can be applied to a processing scenario of a predicted image containing a semi-transparent object. The electronic device identifies and separates the semi-transparent object and the non-transparent object in the image frame. For the semi-transparent object, the mixing coefficient of the semi-transparent object in the two image frames before and after is determined based on its brightness value, so as to perform a mixed prediction of the semi-transparent object in the two image frames before and after, and obtain a first prediction result corresponding to the semi-transparent object. Among them, the mixing coefficient of the first prediction result is obtained based on the brightness value of the non-transparent object in the two image frames before and after, and the mixed prediction result has a higher accuracy and is closer to the normal change state of the two image frames before and after. For the non-transparent object in the image frame, the predicted image frame is processed based on the traditional prediction frame technology to obtain a second prediction result corresponding to the non-transparent object. Further, the electronic device mixes the first prediction result with the second prediction result to obtain a first target predicted image frame. During the image processing process, the electronic device can also obtain a real image frame after rendering and synthesis of the current frame according to the semi-transparent object and the non-transparent object of the drawn current frame. Thereby, the first target predicted image frame is inserted between the real image frame of the current frame and the image frame of the previous frame to achieve an interpolation effect. The electronic device determines the blending coefficient based on the brightness value of the translucent object in the image frame, and separates the translucent objects in the image frame separately and adopts a blending processing method, which can accurately and more closely predict the interpolated frame of the real changes of the two previous and next image frames, thereby avoiding problems such as screen freeze, shadow jitter, special effect loss, etc. caused by the inability to obtain the correct interpolated frame; the target predicted image frame takes into account the display characteristics of translucent objects and non-transparent objects in the image frame, and the target predicted image frame has high accuracy. In the process of displaying the target predicted image frame, the running smoothness of the overall picture is improved, and the power consumption of the electronic device for screen rendering and display can be effectively reduced.

[0090] The image processing method provided in the embodiments of the present application can be applied to electronic devices. For example, the electronic device can be a portable computer (such as a mobile phone), a tablet computer, a laptop computer, a personal computer (PC), a wearable electronic device (such as a smart watch), an augmented reality (AR) or virtual reality (VR) device, a car computer, etc. The following embodiments do not impose any special restrictions on the specific form of the electronic device.

[0091] Please refer to Figure 3, which shows a structural block diagram of an electronic device (such as electronic device 100) provided in an embodiment of the present application. Among them, the electronic device 100 may include a processor 310, an external memory interface 320, an internal memory 321, a universal serial bus (USB) interface 330, a charging management module 340, a power management module 341, a battery 342, an antenna 1, an antenna 2, a radio frequency module 350, a communication module 360, an audio module 370, a speaker 370A, a receiver 370B, a microphone 370C, an earphone interface 370D, a sensor module 380, a button 390, a motor 391, an indicator 392, a camera 393, a display screen 394, and a subscriber identification module (SIM) card interface 395, etc.

[0092] The structure shown in the embodiment of the present invention does not limit the electronic device 100. It may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0093] The processor 310 may include one or more processing units. For example, the processor 310 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0094] The controller can be a decision maker that directs the various components of the electronic device 100 to work in coordination according to the instructions. It is the nerve center and command center of the electronic device 100. The controller generates an operation control signal according to the instruction operation code and timing signal to complete the control of fetching and executing instructions.

[0095] The processor 310 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 310 is a high-speed cache memory, which can store instructions or data that the processor 310 has just used or cyclically used. If the processor 310 needs to use the instruction or data again, it can be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 310, and thus improves the efficiency of the system.

[0096] In some embodiments, the processor 310 may include an interface. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a SIM interface, and / or a USB interface, etc.

[0097] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDL) and a serial clock line (SCL). In some embodiments, the processor 310 may include multiple groups of I2C buses. The processor 310 may be coupled to the touch sensor 380K, the charger, the flash, the camera 393, etc. through different I2C bus interfaces. For example: the processor 310 may be coupled to the touch sensor 380K through the I2C interface, so that the processor 310 communicates with the touch sensor 380K through the I2C bus interface to realize the touch function of the electronic device 100.

[0098] The I2S interface can be used for audio communication. In some embodiments, the processor 310 can include multiple groups of I2S buses. The processor 310 can be coupled to the audio module 370 via the I2S bus to achieve communication between the processor 310 and the audio module 370. In some embodiments, the audio module 370 can transmit an audio signal to the communication module 360 ​​via the I2S interface to achieve the function of answering a call through a Bluetooth headset.

[0099] The PCM interface can also be used for audio communication, sampling, quantizing and encoding analog signals. In some embodiments, the audio module 370 and the communication module 360 ​​can be coupled via a PCM bus interface. In some embodiments, the audio module 370 can also transmit audio signals to the communication module 360 ​​via the PCM interface to realize the function of answering calls via a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication, and the sampling rates of the two interfaces are different.

[0100] The UART interface is a universal serial data bus for asynchronous communication. The bus is a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is generally used to connect the processor 310 and the communication module 360. For example, the processor 310 communicates with the Bluetooth module through the UART interface to implement the Bluetooth function. In some embodiments, the audio module 370 can transmit an audio signal to the communication module 360 ​​through the UART interface to implement the function of playing music through a Bluetooth headset.

[0101] The MIPI interface can be used to connect the processor 310 with peripheral devices such as the display screen 394 and the camera 393. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 310 and the camera 393 communicate via the CSI interface to implement the shooting function of the electronic device 100. The processor 310 and the display screen 394 communicate via the DSI interface to implement the display function of the electronic device 100.

[0102] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 310 with the camera 393, the display screen 394, the communication module 360, the audio module 370, the sensor module 380, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0103] The USB interface 330 may be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 330 may be used to connect a charger to charge the electronic device 100, or may be used to transmit data between the electronic device 100 and a peripheral device. The USB interface 330 may also be used to connect headphones to play audio through the headphones. The USB interface 330 may also be used to connect other electronic devices, such as AR devices, etc.

[0104] The interface connection relationship between the modules shown in the embodiment of the present invention is only for illustrative purposes and does not constitute a structural limitation on the electronic device 100. The electronic device 100 may adopt different interface connection methods in the embodiment of the present invention, or a combination of multiple interface connection methods.

[0105] The charging management module 340 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 340 may receive charging input from a wired charger through the USB interface 330. In some wireless charging embodiments, the charging management module 340 may receive wireless charging input through a wireless charging coil of the electronic device 100. While the charging management module 340 is charging the battery 342, it may also power the electronic device 100 through the power management module 341.

[0106] The power management module 341 is used to connect the battery 342, the charging management module 340 and the processor 310. The power management module 341 receives input from the battery 342 and / or the charging management module 340, and supplies power to the processor 310, the internal memory 321, the external memory interface 320, the display screen 394, the camera 393, and the communication module 360. The power management module 341 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some embodiments, the power management module 341 can also be set in the processor 310. In some embodiments, the power management module 341 and the charging management module 340 can also be set in the same device.

[0107] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the radio frequency module 350, the communication module 360, the modem and the baseband processor.

[0108] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, a cellular network antenna can be reused as a wireless local area network diversity antenna. In some embodiments, the antenna can be used in combination with a tuning switch.

[0109] The RF module 350 can provide a communication processing module for wireless communication solutions including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The RF module 350 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The RF module 350 receives electromagnetic waves from the antenna 1, and performs filtering, amplification, and other processing on the received electromagnetic waves, and transmits them to the modem for demodulation. The RF module 350 can also amplify the signal modulated by the modem, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the RF module 350 can be set in the processor 310. In some embodiments, at least some of the functional modules of the RF module 350 can be set in the same device as at least some of the modules of the processor 310.

[0110] The modem may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 370A, a receiver 370B, etc.), or displays an image or video through a display screen 394. In some embodiments, the modem may be an independent device. In some embodiments, the modem may be independent of the processor 310 and be arranged in the same device as the RF module 350 or other functional modules.

[0111] The communication module 360 ​​can provide a communication processing module for wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The communication module 360 ​​can be one or more devices integrating at least one communication processing module. The communication module 360 ​​receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and filters it, and sends the processed signal to the processor 310. The communication module 360 ​​can also receive the signal to be sent from the processor 310, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0112] In some embodiments, the antenna 1 of the electronic device 100 is coupled to the RF module 350, and the antenna 2 is coupled to the communication module 360, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global satellite positioning system (satellite based augmentation systems, SBAS), a global navigation satellite system (global navigation satellite system, GLONASS), a BeiDou navigation satellite system (BeiDounavigation satellite system, BDS), a Quasi-Zenith satellite system (Quasi-Zenith satellite system, QZSS) and / or a satellite based augmentation system (SBAS).

[0113] The electronic device 100 implements the display function through a GPU, a display screen 394, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 394 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 310 may include one or more GPUs, which execute program instructions to generate or change display information.

[0114] The display screen 394 is used to display images, videos, etc. For example, the display screen 394 can display an incoming call reminder interface and a voice call interface. In an embodiment of the present application, if the electronic device 100 receives an in-application call request initiated by the opposite end in the first application, the display screen 394 of the electronic device 100 can display a voice call interface including the business information of the first application. The display screen 394 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 394, where N is a positive integer greater than 1.

[0115] In some embodiments, the display screen displays image frames based on a preset refresh rate. For example, the refresh rate can be 60Hz, 90Hz, 120Hz, etc. The refresh rate refers to the frequency at which the display screen refreshes image frames. For example, when the refresh rate is 60Hz, it means that the display screen refreshes 60 image frames in one second. In some embodiments, the frame rate of the electronic device may be consistent with the refresh rate of the display screen, or it may be inconsistent. Among them, the frame rate of the electronic device refers to the frequency at which the electronic device draws and renders image frames. For example, when the frame rate is 60Hz, it means that the electronic device draws and renders 60 image frames in one second.

[0116] In some embodiments, there may be a problem of screen display jamming due to inconsistency between the frame rate and refresh rate of the electronic device. For scenes containing translucent objects in the screen, the image processing method provided in the embodiments of the present application can effectively solve the problem of screen jamming. At the same time, it can also avoid the problem of abnormal display of translucent objects on the screen caused by the traditional prediction frame technology being unable to generate correct prediction image frames.

[0117] The electronic device 100 can realize the shooting function through ISP, camera 393, video codec, GPU, display screen and application processor.

[0118] The ISP is used to process the data fed back by the camera 393. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. The ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP can be set in the camera 393.

[0119] The camera 393 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 393, where N is a positive integer greater than 1.

[0120] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0121] Video codecs are used to compress or decompress digital videos. The electronic device 100 may support one or more video codecs. Thus, the electronic device 100 may play or record videos in a variety of coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0122] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of electronic device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0123] The external memory interface 320 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 310 through the external memory interface 320 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0124] The internal memory 321 can be used to store computer executable program codes, which include instructions. The processor 310 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 321. The memory 321 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the memory 321 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, other volatile solid-state storage devices, a universal flash storage (UFS), etc.

[0125] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 370, the speaker 370A, the receiver 370B, the microphone 370C, the headphone jack 370D, and the application processor.

[0126] The audio module 370 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signal. The audio module 370 can also be used to encode and decode audio signals. In some embodiments, the audio module 370 can be arranged in the processor 310, or some functional modules of the audio module 370 can be arranged in the processor 310.

[0127] The speaker 370A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The electronic device 100 can listen to music or listen to a hands-free call through the speaker 370A.

[0128] The receiver 370B, also called a "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device 100 receives a call or voice message, the voice can be received by placing the receiver 370B close to the human ear.

[0129] Microphone 370C, also called "microphone" or "microphone", is used to convert sound signals into audio electrical signals. When making a call or sending a voice message, the user can make a sound by putting his mouth close to microphone 370C to input the sound signal into microphone 370C. The electronic device 100 can be provided with at least one microphone 370C. In some embodiments, the electronic device 100 can be provided with two microphones 370C, which can not only collect sound signals but also realize noise reduction function. In some embodiments, the electronic device 100 can also be provided with three, four or more microphones 370C to realize the collection of sound signals, noise reduction, identification of sound sources, realization of directional recording function, etc.

[0130] The earphone interface 370D is used to connect a wired earphone and can be a USB interface 330, or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0131] The key 390 includes a power key, a volume key, etc. The key 390 may be a mechanical key or a touch key. The electronic device 100 receives the key 390 input and generates a key signal input related to the user settings and function control of the electronic device 100.

[0132] Motor 391 can generate vibration prompts. Motor 391 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. Touch operations acting on different areas of the display screen 394 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0133] Indicator 392 may be an indicator light, which may be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0134] The SIM card interface 395 is used to connect the SIM. The SIM card can be connected to and separated from the electronic device 100 by inserting it into the SIM card interface 395 or pulling it out from the SIM card interface 395. The electronic device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 395 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface 395 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 395 can also be compatible with different types of SIM cards. The SIM card interface 395 can also be compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the electronic device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

[0135] The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. Taking the system as an example, the software structure of the electronic device 100 is exemplified.

[0136] Figure 4 1 is a software structure diagram of the electronic device 100 of an embodiment of the present invention. The layered architecture divides the software into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, The system is divided into four layers, from top to bottom: application layer, application framework layer, Android runtime and system library, and kernel layer.

[0137] The application layer can include a series of application packages.

[0138] like Figure 4 As shown, the application package may include applications such as games, gallery, calendar, phone, map, navigation, WLAN, Bluetooth, music, video, short message, etc.

[0139] In some embodiments, each application includes an application main thread and a rendering thread. The application main thread is used to draw the corresponding image frame when a drawing signal arrives. The rendering thread is used to render the drawn image frame.

[0140] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0141] like Figure 4 As shown, the application framework layer may include a window manager, a content provider, a view system, a telephony manager, a resource manager, and a notification manager.

[0142] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0143] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0144] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0145] The phone manager is used to provide communication functions of the electronic device 100. For example, the management of call status (including answering, hanging up, etc.).

[0146] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0147] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.

[0148] Android runtime includes core library and virtual machine. Android runtime is responsible for scheduling and management of Android system. Core library includes two parts: one is the function that needs to be called by Java language, and the other is Android core library.

[0149] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0150] The system library may include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0151] The surface manager is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications. The media library supports playback and recording of multiple common audio and video formats, as well as static image files. The media library can support multiple audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc. The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, synthesis, and layer processing. The 2D graphics engine is a drawing engine for 2D drawing.

[0152] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

[0153] Optionally, in the scene of rendering, processing, and displaying an image frame including a semi-transparent object, the image processing method provided in this embodiment may be executed by a processing module of an electronic device, and the processing module may be deployed in the kernel layer. Exemplarily, in the scene of rendering and displaying a game animation scene, the processing module may include a game engine.

[0154] In some embodiments, exemplarily, in the rendering and display scene of the game animation scene, the process of drawing, rendering, and sending the image frame in the game animation includes: the application main thread of the game application draws the image frame when the drawing signal arrives; after the application main thread completes the drawing of the image frame, it sends a rendering instruction to the rendering thread; the rendering thread performs a rendering operation on the drawn image frame, and after completing the rendering operation of the image frame, the rendering thread will store the rendered image frame in the corresponding cache queue. The synthesis thread deployed in the kernel layer reads the rendered image frame in the cache queue for synthesis operation, and sends the synthesized image frame to the display driver for display.

[0155] In the above-mentioned process of processing the image frame, the game engine can obtain the drawn image frame by intercepting the rendering instruction. The image processing method provided in this embodiment is executed for the drawn image frame. Specifically, it includes:

[0156] The game engine identifies and separates the semi-transparent objects and non-transparent objects in the acquired drawn image frames, and obtains the semi-transparent objects and non-transparent objects respectively. The game engine performs mixed prediction on the semi-transparent objects based on the brightness values ​​of the semi-transparent objects in the previous frame and the current frame, and obtains the first prediction result corresponding to the semi-transparent objects; the game engine performs prediction based on the traditional prediction frame algorithm based on the non-transparent objects in the previous frame and the current frame, and obtains the second prediction result corresponding to the non-transparent objects. Furthermore, the game engine performs mixed processing on the first prediction result and the second prediction result to obtain the target prediction image frame.

[0157] During the image processing, the game engine can also obtain the real image frame of the current frame based on the semi-transparent objects and non-transparent objects of the current frame, thereby inserting the target predicted image frame between the real image frame of the current frame and the image frame of the previous frame to achieve the interpolation effect.

[0158] In this embodiment, the electronic device can identify and separate translucent objects and non-transparent objects in the image frame, and for the translucent objects, perform mixed prediction based on their brightness values ​​to obtain a first prediction result corresponding to the translucent object, wherein the first prediction result is a mixing coefficient of the previous and next image frames determined based on the brightness value of the non-transparent object. The translucent object in the obtained first prediction result is closer to the change of the translucent object in the previous and next image frames, and has high accuracy, avoiding the problem that the traditional prediction frame technology predicts the entire image frame, but the transparency information of the translucent object in the image frame is lost, resulting in the inability to generate a correct predicted image frame; at the same time, the target predicted image frame is obtained by mixing the first prediction result corresponding to the translucent object and the second prediction result corresponding to the non-transparent object. The target predicted image frame takes into account the display characteristics of the translucent object and the non-transparent object in the image frame. The target predicted image frame has high accuracy. In the process of displaying the target predicted image frame, the picture is smoother, and the technology of predicting image frames can effectively reduce the power consumption of electronic devices displaying image frames.

[0159] It should be noted that any processing scenario of predicted image frames of animations or videos that include translucent objects can use the image processing method provided in the embodiments of the present application to realize the generation and display of predicted image frames, so as to solve the problem of traditional prediction frame technology being unable to generate correct predicted images due to the lack of transparency information of translucent objects, resulting in unsmooth pictures or other problems, thereby reducing the power consumption of electronic devices while improving the display smoothness of electronic devices.

[0160] In some embodiments, taking the game animation scene as an example, illustratively, Figure 5 A flowchart of an image processing method is given, wherein the execution subject is a processing module of an electronic device. In this embodiment, the processing module may be a game engine. The image processing method includes:

[0161] S201, the game engine obtains the current image frame.

[0162] In some embodiments, the game engine can obtain the current image frame by intercepting the rendering instruction of the game screen issued by the main thread of the game application. The current image frame is the image frame that has been drawn and is to be rendered by the main thread of the application in the cycle when the drawing signal arrives. The image frame includes at least a semi-transparent object; or the image frame includes a semi-transparent object and a non-transparent object.

[0163] The current image frame (first image frame) may be the Nth image frame in the drawing and rendering sequence of the current game animation scene, where N is a natural number greater than 1. That is, the current frame may be the second image frame, the third image frame, and so on in the drawing and rendering sequence.

[0164] Among them, a semi-transparent object can be understood as an object with a transparency greater than 0 and less than 1. Correspondingly, an object with a transparency of 0 is a non-transparent object, and an object with a transparency of 1 is a fully transparent object.

[0165] S202: The game engine identifies and separates the semi-transparent objects and non-transparent objects in the current image frame.

[0166] Optionally, in some embodiments, when drawing an image frame, a transparency blending technique (Blend blending) is often used. Blend blending is a technique for mixing a source color and a target color in some way to generate special effects. Blend blending is often used to draw transparent or translucent objects. The blending value (alpha value) that plays a key role in blending is actually to blend the source color and the target color at a given ratio to achieve different degrees of transparency. An alpha value of 0 means completely transparent, and an alpha value of 1 means completely opaque.

[0167] The game engine can identify and separate translucent objects and non-transparent objects in the current image frame according to the alpha value of the objects in the current image frame. For example, objects with an alpha value greater than 0 and less than 1 are considered translucent objects; objects with an alpha value of 0 are considered non-transparent objects.

[0168] In some embodiments, the Open Graphics Library (OpenGL) is used to provide a cross-language, cross-platform application programming interface (API) for rendering two-dimensional and three-dimensional vector graphics. Among them, OpenGL is a state machine. When rendering semi-transparent objects, OpenGL can modify the corresponding rendering state. In order to render different transparency levels, OpenGL will enable the Alpha blending state, and the Alpha blending state is gl Enable (GL_BLEND); when rendering non-transparent objects, OpenGL will turn off the Alpha blending state, and the Alpha blending state is gl Disab le (GL_BLEND).

[0169] In some embodiments, the game engine can determine whether the rendering instruction is for rendering a semi-transparent object or a non-transparent object by using the blending state of the cached GL_BLEND. That is, the semi-transparent object and the non-transparent object in the image frame are identified based on the blending state of GL_BLEND. According to the blending state of GL_BLEND corresponding to each object in the cache, the object with the blending state enabled (gl Enable) is determined as a semi-transparent object; the object with the blending state disabled (gl Disable) is determined as a non-transparent object.

[0170] After the game engine identifies the semi-transparent objects and the non-transparent objects in the current image frame, the semi-transparent objects and the non-transparent objects may be stored separately.

[0171] In some embodiments, the image frame to be rendered has a corresponding cache queue (framebuffer), and the cache queue is used to store the rendered image frame. For example, the cache queue of application 1 is cache queue 1, then the image frame to be rendered indicated by the rendering instruction issued by application 1 will be stored in cache queue 1 after the image rendering is completed, waiting to be synthesized and displayed. For example, the cache queue of application 2 is cache queue 2, then the image frame to be rendered indicated by the rendering instruction issued by application 2 will be stored in cache queue 2 after the image rendering is completed, waiting to be synthesized and displayed.

[0172] Optionally, there is a binding relationship between the image frame and the cache queue. For example, the cache queue bound to the image frame 1 is queue 1. After the image frame 1 is rendered, the image frame 1 can be stored in the cache queue 1 according to the binding relationship of the image frame 1.

[0173] Optionally, in order to more effectively perform image processing on translucent objects and execute subsequent rendering, synthesis, and display processes, the game engine may store the identified translucent objects separately from the non-transparent objects.

[0174] Exemplarily, after identifying the semi-transparent objects and non-transparent objects of the current image frame, the game engine determines that the cache queue bound to the current image frame is the first cache queue according to the binding relationship of the current image frame. For non-transparent objects, before executing the rendering instruction, the game engine keeps the binding relationship between the non-transparent objects and the first cache queue unchanged, and after the non-transparent objects are subsequently rendered, the rendered non-transparent objects can still be stored in the first cache queue.

[0175] For translucent objects, the game engine can create a second cache queue that only stores translucent objects. Before executing the rendering instruction, it obtains the identifier of the first cache queue bound to the current image frame, such as the first ID, and unbinds the binding relationship between the current image frame and the first cache queue according to the first ID, and obtains the identifier of the newly created second cache queue, such as the second ID, and binds the current image frame to the second cache queue according to the second ID. After the semi-transparent objects are subsequently rendered, the rendered semi-transparent objects can be stored in the second cache queue.

[0176] In some embodiments, before executing the rendering instruction of the non-transparent object, if the current image frame is bound to the second cache queue, the game engine can also unbind the current image frame from the second cache queue and bind the current image frame to the first cache queue, so that the non-transparent object is stored in the first cache queue after rendering.

[0177] After the game engine calls the rendering thread to render the non-transparent objects of the current image frame in the first cache queue, the rendering result only includes the non-transparent objects rendered in the current image frame; after the game engine calls the rendering thread to render the semi-transparent objects of the current image frame in the second cache queue, the rendering result only includes the semi-transparent objects rendered in the current image frame.

[0178] After the game engine identifies and separates the semi-transparent objects and the non-transparent objects in the current image frame, the electronic device executes:

[0179] S203: Obtain the semi-transparent object in the current image frame.

[0180] Optionally, the game engine may obtain a semi-transparent object in the current image frame, and determine a region texture1 where the semi-transparent object is located.

[0181] S204: Obtain a semi-transparent object in a previous image frame, and based on brightness values ​​corresponding to the semi-transparent objects in the previous image frame and the current image frame, perform a mixing process on the previous image frame and the current image frame to obtain a first prediction result.

[0182] The previous image frame (the second image frame) is the previous image frame in the time sequence of the current image frame (the first image frame).

[0183] In some embodiments, after the game engine identifies and separates the translucent objects in the current image frame, if the current image frame is directly reused as the result of the interpolated image frame (predicted image frame), it will still cause screen freezes and cannot solve the problem of special effects being out of place.

[0184] In this embodiment, the game engine performs blending processing on the semi-transparent objects in the current image frame and the previous image frame to obtain a more accurate predicted image frame.

[0185] In some embodiments, the game engine may also determine a mixing coefficient based on the brightness values ​​corresponding to each pixel in the semi-transparent object in the current image frame and the previous image frame, thereby calculating the target pixel value corresponding to each pixel based on the mixing coefficient to obtain a first prediction result.

[0186] Exemplarily, the electronic device determines a mixing coefficient according to brightness values ​​corresponding to the semi-transparent objects in the current image frame and the previous image frame, and performs mixing processing on the semi-transparent objects in the current image frame and the previous image frame based on the mixing coefficient to obtain a first prediction result, referring to Figure 6 ,include:

[0187] S2041. The game engine traverses pixels corresponding to the same position of the semi-transparent object area in the current image frame and the semi-transparent object area in the previous image frame.

[0188] The game engine obtains each pixel at the same position in the current image frame and the previous image frame, and determines the brightness value for each pixel. The pixels at the same position in the current image frame and the previous image frame have a corresponding relationship.

[0189] S2042: Determine whether the value of the pixel at the first position is empty.

[0190] S2043: If the pixels corresponding to the first positions in the current image frame and the previous image frame are both empty, output a null value.

[0191] In some embodiments, if the pixels corresponding to the first position in the current image frame and the previous image frame are both empty or the pixel values ​​are both 0, it means that there is no semi-transparent object at this position, and the output of this pixel position is also empty.

[0192] S2044. If the corresponding pixel at the first position of at least one frame between the current image frame and the previous image frame is not empty, the game engine converts the pixel value of the pixel at the first position into a brightness value, and obtains the brightness difference between the pixel at the first position in the current image frame and the previous image frame.

[0193] In some embodiments, if the pixel corresponding to the first position of the current image frame is not empty, or the pixel corresponding to the first position of the previous image frame is not empty, or the pixels corresponding to the first positions of both the current image frame and the previous image frame are not empty, the game engine obtains the pixel value (RGB value) of the pixel corresponding to the first position of the current image frame, and calculates the brightness value of the pixel corresponding to the first position of the current image frame according to a preset mapping relationship; accordingly, the game engine obtains the pixel value (RGB value) of the pixel corresponding to the first position of the previous image frame, and calculates the brightness value of the pixel corresponding to the first position of the previous image frame according to a preset mapping relationship.

[0194] For example, the mapping relationship between the pixel value (RGB value) of a pixel and the brightness value Y can be expressed as:

[0195] Y=R*0.30+G*0.59+B*0.11.

[0196] After obtaining the brightness values ​​of the pixels at the first position in the current image frame and the previous image frame, the game engine calculates the brightness difference between the two.

[0197] S2045: Determine whether the brightness difference is greater than a preset brightness threshold.

[0198] S2046: If the brightness difference is less than or equal to the preset brightness threshold, use the first mixing coefficient to weightedly mix the pixel values ​​at the first position of the current image frame and the previous image frame, and output the target pixel value of the pixel corresponding to the first position.

[0199] The mixing coefficient is used to represent the ratio of the weighting coefficient of the pixel in the current image frame to the weighting coefficient of the pixel in the previous image frame.

[0200] Exemplarily, the first mixing coefficient can be understood as a mixing coefficient that is relatively balanced for the current image frame and the previous image frame. In the first mixing coefficient, the weighting coefficient of the pixels in the current image frame can be the same as the weighting coefficient of the pixels in the previous image frame. For example, the first mixing coefficient can be 0.5:0.5, where 0.5 is the weighting coefficient of the pixels in the current image frame and 0.5 is the weighting coefficient of the pixels in the previous image frame. The target pixel value c of the i-th pixel in the first prediction result is calculated based on the first mixing coefficient. i , which can be expressed as:

[0201]

[0202] Among them, a i is the pixel value of the i-th pixel in the current image frame, b i is the sum of the previous image frame and a i The pixel value of the corresponding i-th pixel.

[0203] S2047: If the brightness difference is greater than a preset brightness threshold, a second mixing coefficient is used to perform weighted mixing on the pixel values ​​at the first position of the current image frame and the previous image frame, and a target pixel value of the pixel corresponding to the first position is output.

[0204] Exemplarily, the second mixing coefficient can be understood as a mixing coefficient with a weight biased toward the current image frame. In the second mixing coefficient, the weighting coefficient of the pixel in the current image frame is greater than the weighting coefficient of the pixel in the previous image frame. For example, the second mixing coefficient can be 0.8:0.2, where 0.8 is the weighting coefficient of the pixel in the current image frame, and 0.2 is the weighting coefficient of the pixel in the previous image frame. The target pixel value c of the i-th pixel in the first prediction result based on the second mixing coefficient is i , which can be expressed as:

[0205] c i =0.8a i +0.2b i .

[0206] Among them, a i is the pixel value of the i-th pixel in the current image frame, b i is the sum of the previous image frame and a i The pixel value of the corresponding i-th pixel.

[0207] The second mixing coefficient may also be 0.7:0.3, which can be determined according to actual conditions.

[0208] S2048. Obtain a first prediction result based on target pixel values ​​of pixels at all positions in the semi-transparent object area in the current image frame and the previous image frame.

[0209] In some embodiments, after the game engine traverses the pixels at all positions of the semi-transparent object area in the current image frame and the previous image frame, it obtains the first prediction result based on the target pixel values ​​of the pixels at all positions of the semi-transparent object area.

[0210] The first prediction result represents a predicted semi-transparent object region, and the predicted semi-transparent object region has the same size and format as the semi-transparent object region of the current image frame and the semi-transparent object region of the previous image frame.

[0211] Exemplarily, in some embodiments, the above steps S2041-S2045 can also be implemented by constructing a hybrid model (mixer). Exemplarily, the input of the mixer is texture1 of the area where the semi-transparent object of the current image frame is located and texture2 of the area where the semi-transparent object of the previous image frame is located. The size of texture1 of the area where the semi-transparent object of the current image frame is the same as that of texture2 of the area where the semi-transparent object of the previous image frame is located. After the mixer traverses the pixels at all positions in texture1 of the current image frame and texture2 of the previous image frame, the mixer outputs a first prediction result, which is a newly created texture3 of the area where the semi-transparent object is located, which has the same size and format as the input.

[0212] In some embodiments, the mixer can be implemented through a shader program, using the parallel acceleration processing capability of the GPU to improve the execution efficiency of the mixer.

[0213] General mixers use fixed mixing coefficients to mix colors, and the mixed results will have a strong sense of blur. In this embodiment, a mixing processing method based on brightness difference is introduced. The mixing coefficient is adjusted according to the brightness difference of each pixel in the input image frame to weaken the blur of the mixed result.

[0214] S205: Obtain non-transparent objects in the current image frame.

[0215] Optionally, the game engine may obtain a non-transparent object in the current image frame, and determine a region texture4 where the non-transparent object is located.

[0216] S206: Obtain a non-transparent object in a previous image frame, and based on the non-transparent objects in the previous image frame and the current image frame, use a prediction frame algorithm to obtain a second prediction result.

[0217] The second prediction result is the predicted non-transparent object area.

[0218] In some embodiments, for non-transparent objects, the game engine may use conventional prediction frame technology to obtain the second prediction result. The conventional prediction frame technology may be FRC technology, MEMC technology, etc.

[0219] In this embodiment, the game engine can obtain the second prediction result based on the MEMC technology. Figure 7 As shown, Figure 7A game scene is given, which includes a background including a tree, a character, and a translucent object "tail effect" 110. When the character is stationary or moving, the "tail effect" 110 will swing up and down. The game engine uses MEMC technology (such as block matching algorithm) to perform motion estimation based on the previous image frame and the current image frame, obtains the motion vector from the previous image frame to the current image frame, and generates an interpolated image frame (predictive frame) through motion compensation technology based on the current image frame and the motion vector.

[0220] After obtaining the first prediction result and the second prediction result based on the above S204 and S205, the electronic device may obtain a target prediction image frame based on the first prediction result and the second prediction result, including:

[0221] S207 , performing mixed rendering on the first prediction result and the second prediction image frame to obtain a first target prediction image frame.

[0222] In some embodiments, the game engine obtains a first prediction result corresponding to the translucent object and a second prediction result corresponding to the non-transparent object. Based on the transparency blending coefficient (Alpha value) of the translucent object, the game engine can call the rendering thread and use the Alpha blending rendering method to mix the first prediction result and the second prediction result to obtain a first target predicted image frame.

[0223] The principle of alpha blending rendering is to look at non-transparent object A through translucent object B, and the apparent color C is the mixed color of B and A.

[0224] For example, the transparency of the semi-transparent object B is alpha, and the alpha value is (0,1). When the alpha value is 0, it is a completely transparent object; when the alpha value is 1, it is a non-transparent object.

[0225] The R(C), G(C), and B(C) of the mixed image frame can be expressed as:

[0226] R(C)=alpha*R(B)+(1-alpha)*R(A);

[0227] G(C)=alpha*G(B)+(1-alpha)*G(A);

[0228] B(C)=alpha*B(B)+(1-alpha)*B(A).

[0229] Among them, R(x), G(x), and B(x) refer to the RGB components of color x respectively.

[0230] In this way, the game engine can obtain the RGB value of each pixel in the mixed image frame, thereby obtaining the target predicted image frame.

[0231] S208: The game engine obtains a real image frame according to the semi-transparent objects and non-transparent objects in the current image frame.

[0232] In some embodiments, after the game engine identifies and separates the semi-transparent objects and non-transparent objects in the current image frame, it can also generate a real image frame based on the identified semi-transparent objects and non-transparent objects. Exemplarily, the game engine calls the rendering thread and uses the Alpha blending rendering method to blend the identified semi-transparent objects with the non-transparent objects to obtain a real image frame.

[0233] After obtaining the real image frame and the target predicted image frame, the game engine inserts the target predicted image frame between the previous image frame and the real image frame to achieve the purpose of interpolating the predicted frame. Since the target predicted image frame after image processing takes into account the display characteristics of semi-transparent objects and non-transparent objects in the image frame, the target predicted image frame has a higher accuracy. In the process of displaying the target predicted image frame, the picture is smoother, and the technology of predicting image frames can effectively reduce the power consumption of electronic devices displaying image frames.

[0234] In some embodiments, Figure 8 A more detailed flowchart of the image processing method is given, including:

[0235] S301: The game engine obtains the current image frame.

[0236] S302: Identify the corresponding enabling status of the drawn objects in the current image frame.

[0237] S303: If the enabling state of GL_BLEND is yes, determine that the drawn object is a semi-transparent object, and obtain the semi-transparent object.

[0238] S304: If the enabling state of GL_BLEND is no, determining that the drawn object is a non-transparent object, and acquiring the non-transparent object.

[0239] S305: The game engine obtains the area where the semi-transparent object of the previous image frame is located.

[0240] S306: The game engine inputs the area where the semi-transparent object of the previous image frame is located and the area where the semi-transparent object of the current image frame is located into a brightness difference mixer to obtain a first prediction result.

[0241] S307: The game engine obtains the non-transparent object in the previous image frame.

[0242] S308. The game engine uses the MEMC technology to obtain a second prediction result according to the non-transparent objects in the previous image frame and the non-transparent objects in the current image frame.

[0243] S309: The game engine performs Alpha blending rendering on the first prediction result and the second prediction result, and outputs a first target prediction image frame.

[0244] S310: The game engine performs Alpha blending rendering on the semi-transparent objects in the current image frame and the non-transparent objects in the current image frame to obtain a real image frame.

[0245] In some embodiments, when the color of some translucent objects included in the scene rendering is very light (brightness is less than a preset brightness threshold) or the proportion of the screen space occupied by the translucent objects is less than a preset threshold, and the translucent objects do not have a great impact on the scene rendering picture, in order to further reduce the image processing power consumption of the electronic device, there is no need to perform separate processing of the translucent objects, and the real image frame can be directly used for MEMC prediction. If the color of the translucent object is bright or the proportion of the screen space occupied by the translucent object is large, the game engine performs separate processing with the translucent object, that is, executes S204 in the above embodiment.

[0246] In some embodiments, after obtaining the translucent objects in the current image frame, the game engine may make a decision on whether to perform local blending on the translucent objects. If it is determined that the translucent objects in the current image frame meet the preset conditions, local blending is performed; if it is determined that the translucent objects in the current image frame do not meet the preset conditions, no local blending is performed. Exemplarily, the preset conditions may include that the proportion of the screen space occupied by the translucent objects is greater than a preset threshold. The preset threshold may be 30%, 40%, etc.

[0247] In some embodiments, the game engine determines whether the proportion of the screen space occupied by the translucent object is greater than a preset threshold, and can convert the value of the pixel in the area where the translucent object is located into brightness, for example, normalize the pixel value to 0.0-1.0 as the brightness value of the pixel. When the brightness is greater than the preset brightness threshold (for example, the preset brightness threshold is 0.5), the pixel is judged as a valid pixel, otherwise it is an invalid pixel. Then the number of valid pixels is counted and divided by the total number of pixels on the screen to obtain the effective proportion value. When the effective proportion value is greater than the preset threshold (for example, the preset threshold is 30%), the output decision result is "local mixing processing is required", otherwise the output decision result is "local mixing processing is not required".

[0248] For example, Fig. 9A flowchart of an image processing method is given, including:

[0249] S201, the game engine obtains the current image frame.

[0250] S202: The game engine identifies and separates the semi-transparent objects and non-transparent objects in the current image frame.

[0251] S203: Obtain the semi-transparent object in the current image frame.

[0252] S209: The game engine inputs the semi-transparent object of the current image frame into the local blending decision to obtain the local blending decision result.

[0253] In some embodiments, if the translucent object in the current image frame meets the preset conditions, the game engine outputs a decision result of "local blending processing is required"; if the translucent object in the current image frame does not meet the preset conditions, the game engine outputs a decision result of "local blending processing is not required".

[0254] S204. When the local blending decision result is used to represent the need to perform local blending processing, the game engine obtains the semi-transparent object of the previous image frame, and based on the brightness values ​​corresponding to the semi-transparent objects in the previous image frame and the current image frame, blends the previous image frame with the current image frame to obtain a first prediction result.

[0255] S205: Obtain non-transparent objects in the current image frame.

[0256] S206. When the local blending decision result is used to represent the need to perform local blending processing, the game engine obtains the non-transparent objects in the previous image frame, and based on the non-transparent objects in the previous image frame and the current image frame, uses the MEMC frame algorithm to obtain a second prediction result.

[0257] S207 . Mix and render the first prediction result and the second prediction result to obtain a first target prediction image frame.

[0258] S208: The game engine obtains a real image frame according to the semi-transparent objects and non-transparent objects in the current image frame.

[0259] S210: When the local mixing decision result is used to indicate that the local mixing process does not need to be performed, the game engine obtains the previous image frame, and based on the previous image frame and the real image frame, uses the MEMC algorithm to obtain a second target predicted image frame.

[0260] In some embodiments, when the local blending decision result is used to represent that local blending processing does not need to be performed, the game engine can ignore the translucent objects in the image frame and use traditional prediction frame technology (such as MEMC technology) to combine the previous image frame with the acquired real image frame to obtain a predicted image frame.

[0261] In this embodiment, after acquiring the current image frame, the game engine identifies the translucent object in the current image frame to determine whether it meets the preset conditions. If the preset conditions are met, that is, the changes in the translucent objects in the current image frame can cause relatively obvious perception to the user, S202-S208 are executed to acquire the target predicted image frame based on the image processing method provided in this embodiment, which can ensure the smoothness of the picture while reducing the power consumption of the electronic device rendering based on the predicted frame technology; if the translucent object in the current image frame does not meet the preset conditions, that is, the changes in the translucent objects in the current image frame will not cause obvious perception to the user, S210 is executed to acquire the predicted image frame using the traditional predicted frame technology, which can further reduce the computing power consumption of the electronic device.

[0262] Fig.10 A possible structural diagram of the electronic device involved in the above embodiments is shown. Fig.10 The electronic device 1000 shown includes a processor 1001 , a display screen 1002 , and a storage module 1003 .

[0263] Among them, the processor 1001 can be a central processing unit (CPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The processor may include an application processor and a baseband processor. It can implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. The processor may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like. The storage module 1003 may be a memory.

[0264] For example, the processor 1001 may be Figure 3 The processor 310 shown; the display screen 1002 can be as shown Figure 3 The display screen 394 shown; the storage module 1003 can be as follows Figure 3The internal memory 321 shown. The electronic device provided in the embodiment of the present application can be Figure 3 An electronic device 100 is shown.

[0265] The present application also provides a chip system (eg, a system on a chip (SoC)), such as Fig.11 As shown, the chip system includes at least one processor 701 and at least one interface circuit 702. The processor 701 and the interface circuit 702 can be interconnected by lines. For example, the interface circuit 702 can be used to receive signals from other devices (such as a memory of an electronic device). For another example, the interface circuit 702 can be used to send signals to other devices (such as a processor 701 or a camera of an electronic device). Exemplarily, the interface circuit 702 can read instructions stored in the memory and send the instructions to the processor 701. When the instruction is executed by the processor 701, the electronic device can perform the various steps in the above embodiments. Of course, the chip system may also include other discrete devices, which are not specifically limited in the embodiments of the present application.

[0266] An embodiment of the present application also provides a computer-readable storage medium, which includes computer instructions. When the computer instructions are executed on the above-mentioned electronic device, the electronic device executes each function or step executed by the electronic device 100 in the above-mentioned method embodiment.

[0267] The embodiment of the present application further provides a computer program product, which, when executed on a computer, enables the computer to execute the functions or steps executed by the electronic device 100 in the above method embodiment. For example, the computer may be the above electronic device 100.

[0268] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0269] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0270] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0271] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0272] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0273] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application shall be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. An image processing method, characterized in that: The method comprises: The electronic device acquires a first image frame; the first image frame is the Nth drawn but not rendered image frame in the drawing and rendering scene in time sequence, wherein N is a natural number greater than 1; the first image frame includes a semi-transparent object and a non-transparent object; the transparency of the semi-transparent object is greater than 0 and less than 1, and the transparency of the non-transparent object is 1; The electronic device performs recognition processing on the first image frame to obtain translucent objects and non-transparent objects in the first image frame; The electronic device acquires a second image frame; the second image frame is a previous image frame of the first image frame in time sequence; The electronic device performs mixed prediction processing on the semi-transparent object in the second image frame and the semi-transparent object in the first image frame to obtain a first prediction result; the first prediction result represents a semi-transparent object area after the prediction mixture; The electronic device performs prediction processing on the non-transparent object in the second image frame and the non-transparent object in the first image frame to obtain a second prediction result; the second prediction result represents the predicted non-transparent object area; The electronic device performs a mixed process on the first prediction result and the second prediction result to obtain a first target prediction image frame; The electronic device performs rendering synthesis according to the semi-transparent objects and non-transparent objects in the first image frame to obtain a real image frame corresponding to the first image frame; The electronic device displays the first target predicted image frame between the second image frame and the real image frame.

2. The method according to claim 1, characterized in that The first image frame includes a plurality of objects; the electronic device performs recognition processing on the first image frame to obtain translucent objects and non-transparent objects in the first image frame, including: The electronic device identifies a mixing state corresponding to each of the objects in the first image frame; the mixing state is used to indicate whether the object has been mixed; The objects whose blending state is enabled are determined as the semi-transparent objects, and the objects whose blending state is disabled are determined as the non-transparent objects.

3. The method according to claim 2, characterized in that The first image frame corresponds to a first cache queue, and the first cache queue is used to store the rendered first image frame; the method further includes: The electronic device creates a second cache queue; The electronic device stores the rendered semi-transparent objects in the first image frame in the second cache queue.

4. The method according to claim 3, characterized in that The method further comprises: The electronic device stores the rendered non-transparent objects in the first image frame in the first cache queue.

5. The method according to any one of claims 1 to 4, characterized in that The electronic device performs mixed prediction processing on the semi-transparent object in the second image frame and the semi-transparent object in the first image frame to obtain a first prediction result, including: The electronic device determines a mixing coefficient based on a brightness difference between a first pixel included in a semi-transparent object region in the first image frame and a second pixel included in a semi-transparent object region in the second image frame; pixels at the same position of the first image frame and the second image frame have a corresponding relationship; the mixing coefficient is used to represent a ratio of a weighting coefficient of the first pixel to a weighting coefficient of the second pixel; The electronic device acquires a pixel value of a third pixel based on the mixing coefficient, the pixel value of the first pixel, and the pixel value of the second pixel; Based on the pixel value of the third pixel, the first prediction result is obtained; the semi-transparent body area represented by the first prediction result is the same size as the semi-transparent object area in the first image frame and the semi-transparent object area in the second image frame.

6. The method according to claim 5, characterized in that The electronic device determines the mixing coefficient based on the brightness difference between the first pixel included in the area where the semi-transparent object is located in the first image frame and the second pixel included in the area where the semi-transparent object is located in the second image frame, including: The electronic device obtains a brightness difference between an i-th first pixel and an i-th second pixel; a position of the i-th first pixel in the semi-transparent object region in the first image frame is the same as a position of the i-th second pixel in the semi-transparent object region in the second image frame; If the brightness difference is less than or equal to the preset difference, the mixing coefficient is a first mixing coefficient; in the first mixing coefficient, the weighting coefficient of the first pixel is the same as the weighting coefficient of the second pixel; If the brightness difference is greater than the preset difference, the mixing coefficient is a second mixing coefficient, and a weighting coefficient of the first pixel in the second mixing coefficient is greater than a weighting coefficient of the second pixel.

7. The method according to claim 6, characterized in that The electronic device acquires a pixel value of a third pixel based on the mixing coefficient, the pixel value of the first pixel, and the pixel value of the second pixel, including: The electronic device performs weighted processing on the pixel value of the i-th first pixel and the pixel value of the i-th second pixel based on the mixing coefficient to obtain the pixel value of the i-th third pixel.

8. The method according to claim 5, characterized in that The method further comprises: If the pixel values ​​of the j-th first pixel and the j-th second pixel are both empty, the electronic device determines that the pixel value of the j-th pixel in the first prediction result is an empty value; the position of the j-th first pixel in the semi-transparent object area in the first image frame, the position of the j-th second pixel in the semi-transparent object area in the second image frame, and the position of the j-th pixel in the semi-transparent object area of ​​the first prediction result are the same.

9. The method according to any one of claims 6 to 8, characterized in that: The method further comprises: The electronic device obtains the brightness value of the i-th first pixel according to the pixel value of the i-th first pixel and the mapping relationship; the pixel value includes an RGB value, and the mapping relationship includes a conversion relationship between the RGB value of the pixel and the brightness value; as well as, The electronic device obtains the brightness value of the i-th second pixel according to the pixel values ​​of the i-th second pixels and the mapping relationship.

10. The method according to any one of claims 1 to 4, characterized in that The electronic device performs a mixed process on the first prediction result and the second prediction result to obtain a first target prediction image frame, including: The electronic device uses an Alpha blending rendering method to blend the first prediction result with the second prediction result according to the transparency of the semi-transparent object in the first prediction result, and obtains a blended pixel value of each pixel after the blending process; The electronic device obtains the first target predicted image frame according to the mixed pixel values ​​of the pixels.

11. The method according to any one of claims 1 to 4, characterized in that: The electronic device performs prediction processing on the non-transparent object in the second image frame and the non-transparent object in the first image frame to obtain a second prediction result, including: The electronic device uses motion estimation and compensation MEMC technology to predict the non-transparent objects in the second image frame and the non-transparent objects in the first image frame to obtain the second prediction result.

12. The method according to any one of claims 1 to 4, characterized in that After acquiring the semi-transparent object and the non-transparent object in the first image frame, the method further includes: The electronic device determines a mixed decision result according to the semi-transparent object in the first image frame and a preset condition; If the semi-transparent object in the first image frame satisfies any one of the preset conditions, the blending decision result is to perform blending processing; If the semi-transparent object in the first image frame does not meet the preset condition, the blending decision result is not to perform blending processing, Among them, the preset condition includes that the number of valid pixels in the area where the semi-transparent object is located accounts for a proportion of the total number of pixels on the screen of the electronic device that is greater than a preset threshold, and the valid pixels are pixels whose brightness values ​​are greater than the brightness threshold.

13. The method according to claim 12, characterized in that The electronic device performs mixed prediction processing on the semi-transparent object in the second image frame and the semi-transparent object in the first image frame to obtain a first prediction result, including: When the hybrid decision result is to perform hybrid processing, the electronic device performs hybrid prediction processing on the semi-transparent object in the second image frame and the semi-transparent object in the first image frame to obtain the first prediction result.

14. An electronic device, characterized in that: The electronic device includes a memory, a display screen and one or more processors; the memory and the display screen are coupled to the processor; the memory stores computer program code, and the computer program code includes computer instructions, and when the computer instructions are executed by the processor, the electronic device executes the method as described in any one of claims 1-13.

15. A computer-readable storage medium, characterized in that: The method comprises computer instructions, which, when executed on an electronic device, cause the electronic device to execute the method as claimed in any one of claims 1 to 13.

Images