Image processing method and device

Abstract

The invention provides an image processing method and device which are applied to electronic equipment, and the method comprises the steps: obtaining a first main image from a first real image and obtaining a second main image from a second real image, the first real image and the second real image being images rendered by the electronic equipment; performing motion vector calculation on the first main image and the second main image to obtain a motion vector; according to the motion vector, the environment in the target main image and the motion object in the target main image are distorted, the environment in the prediction frame and the motion object in the prediction frame are generated, and the target main image is one frame of main image in the first main image and the second main image; and multiplexing image data of the shadow of the moving object in the target main image in the prediction frame to generate the shadow of the moving object in the prediction frame. Therefore, the shadow jitter problem of the prediction frame is reduced, the difference between the prediction frame and the real image is reduced, and the user experience is improved.

Description

Image processing method and device

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on November 21, 2022, with application number 202211454684.3 and invention name “Image Processing Method and Device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of image processing technology, and in particular to an image processing method and device. Background Art

[0003] Predictive frame technology is a mainstream technique in image rendering. Electronic devices use it to reduce rendering overhead. However, to enhance user immersion, electronic devices incorporate shadow rendering. This technology introduces shadow jitter in the predicted frames, increasing the difference between the predicted frames and the real image, significantly reducing the user experience. The real image is generated by the electronic device based on its rendering logic, and the real image is used to generate the predicted frame.

[0004] Summary of the Invention

[0005] This application provides an image processing method and device, the purpose of which is to solve the problem of shadow jitter in predicted frames, reduce the difference between predicted frames and real images, and improve user experience. To achieve the above purpose, this application provides the following technical solutions:

[0006] In a first aspect, the present application provides an image processing method for use in an electronic device. The method comprises: obtaining a first main image from a first real image and obtaining a second main image from a second real image, wherein the first real image and the second real image are images rendered by the electronic device; calculating motion vectors for the first main image and the second main image to obtain motion vectors; warping the environment and the moving object in the target main image based on the motion vectors to generate the environment and the moving object in a predicted frame, wherein the target main image is a frame of the first main image and the second main image; and reusing image data of the shadow of the moving object in the target main image in the predicted frame to generate the shadow of the moving object in the predicted frame. Thus, the shadow of the moving object in the predicted frame can be extracted from the environment in the predicted frame, allowing the shadow of the moving object in the predicted frame to be generated independently. Furthermore, the shadow of the moving object in the predicted frame no longer relies on the motion vectors, but is generated by reusing the image data of the shadow of the moving object in the target main image in the predicted frame. This reduces the problem of shadow jitter in the predicted frame, reduces the difference between the predicted frame and the real image, and improves the user experience.

[0007] In some examples, warping the environment and the moving object in a target main image based on a motion vector to generate the environment and the moving object in a predicted frame, and reusing image data of the moving object's shadow in the target main image in the predicted frame to generate the moving object's shadow in the predicted frame includes: warping the environment and the moving object in the target main image based on the motion vector to generate a first intermediate image, the first intermediate image including the environment and the moving object; and reusing image data of the moving object's shadow in the target main image in the first intermediate image, such that the first intermediate image includes the environment, the moving object, and the moving object's shadow. After obtaining the first intermediate image including the environment, the moving object, and the moving object's shadow, image completion is performed on the first intermediate image to obtain a second intermediate image; and then post-processing is performed on the second intermediate image to fuse the user interface image corresponding to the target main image with the processed second intermediate image to obtain the predicted frame. Because the moving object and its shadow remain unchanged after obtaining the first intermediate image, the moving object in the first intermediate image can be the moving object in the predicted frame, and the moving object's shadow in the first intermediate image can be the moving object in the predicted frame. The environment in the first intermediate image is completed during the image completion process, so the predicted frame includes the completed environment in the first intermediate image, that is, the environment in the second intermediate image.

[0008] In one possible implementation, reusing image data of a moving object's shadow in a target main image in a predicted frame includes: for each pixel in the predicted frame, determining whether the pixel is located in the moving object's shadow in the target main image based on the position of the moving object's shadow in the target main image; when determining that the pixel is located in the moving object's shadow in the target main image, reusing the pixel value of the pixel in the target main image, with the pixel value of the pixel in the target main image being the image data of the pixel in the target main image; and when determining that the pixel is not located in the moving object's shadow in the target main image, maintaining the pixel value of the pixel unchanged. Thus, for a pixel in the predicted frame located in the moving object's shadow in the target main image, the pixel value of the pixel in the target main image is reused, thereby reusing both the position of the moving object's shadow in the target main image and the image data of the moving object's shadow in the target main image, thereby fusing the moving object's shadow in the target main image into the predicted frame. This avoids continuing to use the motion vector used by the environment to generate the moving object's shadow in the predicted frame, reduces shadow jitter in the predicted frame, reduces the difference between the predicted frame and the real image, and improves the user experience. Because the predicted frame reuses the position of the shadow of the moving object in the target main image, the position of the shadow of the moving object in the predicted frame is the same as its position in the target main image. When the image data of the shadow of the moving object in the target main image is reused, the shadow of the moving object has the same shape, color, and brightness in the predicted frame and the target main image.

[0009] In one possible implementation, a motion vector is used to record the movement speed and direction of each pixel in a target main image. Based on the motion vector, the environment and the moving object in the target main image are distorted to generate the environment and the moving object in the predicted frame. The method includes: moving each pixel in the target main image based on the movement speed and direction of each pixel in the target main image; determining whether the pixel is located in the shadow of the moving object in the predicted frame based on the position of the shadow of the moving object in the target main image during the pixel movement; setting the pixel value of the pixel to a preset value if the pixel is determined to be located in the shadow of the moving object in the predicted frame; and setting the pixel value of the pixel to the pixel value of the pixel in the target main image if the pixel is determined not to be located in the shadow of the moving object in the predicted frame. Thus, the preset value is used to distinguish the shadow of the moving object in the predicted frame from other objects in the predicted frame. The preset value is used to remove the shadow of the moving object in the predicted frame and to mark the position of the shadow of the moving object, thereby reserving the shadow of the moving object for the predicted frame, so that the predicted frame can reuse image data of the shadow of the moving object in the target main image.

[0010] In a possible implementation, the preset value includes a preset transparency, and the preset transparency is different from the transparency of other pixels in the target main image, and is distinguished by the preset transparency.

[0011] In one possible implementation, the process of determining the position of the shadow of the moving object in the target main image includes: comparing each to-be-compared pixel in the same position in the environment shadow texture image and the moving object shadow texture image, wherein the environment shadow texture image is generated based on image data of the environment shadow in the target main image, and the moving object shadow texture image is generated based on image data of the moving object shadow in the target main image; upon determining that the to-be-compared pixel is in the shadow region of the moving object but not in the environment shadow, marking the to-be-compared pixel as being in the shadow region of the moving object; upon determining that the to-be-compared pixel is in the shadow region of the moving object and in the environment shadow, marking the to-be-compared pixel as not being in the shadow region of the moving object; and determining the position of the shadow of the moving object in the target main image based on the to-be-compared pixel marked as being in the shadow region of the moving object, wherein the image data of the shadow of the moving object in the target main image includes the pixel value of the to-be-compared pixel located in the position in the target main image. In some examples, the image data of the shadow of the moving object in the target main image is the same as the image data of the shadow of the moving object in the target main image. Through the environment shadow texture image and the moving object shadow texture image, the position of the moving object shadow in the target main image can be determined according to the principle that "when the moving object shadow is located in the environment shadow, the moving object shadow is not presented in the target main image", so as to prepare for determining the position of the moving object shadow in the predicted frame, thereby enabling the reuse of the image data of the moving object shadow.

[0012] In one possible implementation, if the pixel value of the pixel to be compared in the moving object shadow texture image is greater than the pixel value of the pixel to be compared in the environment shadow texture image, it is determined that the pixel to be compared is in the shadow area of ​​the moving object but not in the environment shadow; if the pixel value of the pixel to be compared in the moving object shadow texture image is less than or equal to the pixel value of the pixel to be compared in the environment shadow texture image, it is determined that the pixel to be compared is in the shadow area of ​​the moving object and in the environment shadow.

[0013] In one possible implementation, the pixel value of the pixel to be compared includes at least one of the R channel value, the G channel value, and the B channel value of the pixel to be compared. In some examples, the pixel value of the pixel to be compared includes one of the R channel value, the G channel value, and the B channel value, which can save storage space; in some examples, the pixel value of the pixel to be compared includes two or three of the R channel value, the G channel value, and the B channel value. Then, when comparing the pixel values, the pixel values ​​of the pixel to be compared can be first fused (such as weighted averaging) to obtain a numerical value, and then the numerical values ​​are compared, thereby comprehensively considering the influence of different channel values.

[0014] In one possible implementation, before obtaining the first main image from the first real image and obtaining the second main image from the second real image, the method further includes: when rendering the real image corresponding to the target main image, obtaining image data of the environmental shadow and image data of the shadow of the moving object, the image data of the environmental shadow and the image data of the shadow of the moving object being used to determine the position of the shadow of the moving object in the target main image, and the position of the shadow of the moving object in the target main image and the image data of the shadow of the moving object being used to obtain image data of the shadow of the moving object in the target main image to be reused. Thus, during the process of rendering the real image corresponding to the target main image, the image data of the environmental shadow and the image data of the shadow of the moving object are intercepted to prepare for reuse.

[0015] In one possible implementation, when rendering a real image corresponding to a target main image, obtaining image data of an ambient shadow and image data of a shadow of a moving object includes: rendering the ambient shadow of the real image to generate image data of the ambient shadow, and storing the image data of the ambient shadow in a first frame buffer; creating a second frame buffer and a third frame buffer; copying the image data of the ambient shadow stored in the first frame buffer to the second frame buffer; changing the frame buffer bound to the real image from the first frame buffer to the third frame buffer; and rendering the shadow of a moving object in the real image to generate image data of the shadow of the moving object, and storing the image data of the shadow of the moving object in the third frame buffer. Thus, the image data of the ambient shadow and the image data of the shadow of the moving object are stored in different frame buffers, preventing the two parts of image data from overlapping and becoming difficult to distinguish, thereby facilitating the use of the image data of the ambient shadow and the image data of the shadow of the moving object.

[0016] In one possible implementation, when rendering a real image corresponding to a target main image, obtaining image data of environmental shadows and image data of shadows of moving objects further includes: after rendering the shadows of the moving objects in the real image, changing the frame buffer bound to the real image from the third frame buffer to the first frame buffer; and rendering the shadows of the moving objects in the real image again to generate image data of the shadows of the moving objects, wherein the image data of the shadows of the moving objects is stored in the first frame buffer. While intercepting the image data of environmental shadows and image data of shadows of moving objects, the image data of environmental shadows and image data of shadows of moving objects can be stored in the first frame buffer bound to the real image, thereby ensuring the accuracy of the image data stored in the first frame buffer and preventing the real image (primarily the target main image) from lacking shadows.

[0017] In a possible implementation, copying the image data of the ambient shadow stored in the first frame buffer to the second frame buffer includes: calling a glBlitFramebuffer function to copy the image data of the ambient shadow stored in the first frame buffer to the second frame buffer.

[0018] In a possible implementation, changing the frame buffer to which the real image is bound from the first frame buffer to the third frame buffer includes: calling a glBindFrameBuffer function to change the frame buffer to which the real image is bound from the first frame buffer to the third frame buffer.

[0019] In a possible implementation manner, the method further includes: after generating the shadow of the moving object in the predicted frame, performing image completion on the predicted frame.

[0020] In a second aspect, the present application provides an electronic device comprising: one or more processors and a memory; the memory is used to store computer program code, the computer program code includes computer instructions, and when the one or more processors execute the computer instructions, the electronic device executes the image processing method described in the first aspect.

[0021] In a third aspect, the present application provides a computer-readable storage medium for storing a computer program. When the computer program is executed, it is specifically used to implement the image processing method as described in the first aspect.

[0022] Based on the implementation methods provided in the above aspects, this application can also be further combined to provide more implementation methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] FIG1 is a schematic diagram of generating a prediction frame provided by the present application;

[0024] FIG2 is a hardware structure diagram of the electronic device provided by this application;

[0025] FIG3 is a flowchart of generating a real image provided by the present application;

[0026] FIG4 is a flowchart of generating a prediction frame provided by the present application;

[0027] FIG5 is a flowchart of obtaining image data of environmental shadows and image data of shadows of moving objects in a real image provided by the present application.

[0028] FIG6 is a flowchart of determining the position of the shadow of a moving object in a real image provided by the present application;

[0029] FIG7 is a flowchart of removing the shadow of a moving object generated according to a motion vector provided by the present application;

[0030] FIG8 is a flowchart of the present application for multiplexing image data of shadows of moving objects in real images;

[0031] FIG9 is a schematic diagram of multiplexed shadows provided by this application. DETAILED DESCRIPTION

[0032] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. The terms used in the following embodiments are only for the purpose of describing specific embodiments and are not intended to be limiting of the present application. As used in the specification and appended claims of the present application, the singular expressions "one", "a kind of", "said", "above", "the" and "this" are intended to also include expressions such as "one or more", unless there is a clear contrary indication in the context. It should also be understood that in the embodiments of the present application, "one or more" refers to one, two or more; "and / or" describes the association relationship of associated objects, indicating that three relationships may exist; for example, A and / or B can represent: the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the related objects before and after are in an "or" relationship.

[0033] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present application. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in yet other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0034] The "multiple" involved in the embodiments of the present application means greater than or equal to two. It should be noted that in the description of the embodiments of the present application, the words "first" and "second" are only used for the purpose of distinguishing the description and cannot be understood as indicating or implying relative importance or order.

[0035] When a user uses an electronic device, the electronic device can display multimedia stream files to the user through a display screen, so as to provide the user with a rich visual experience. Take the multimedia stream as a video stream as an example. The video stream may include multiple frames of images. Exemplarily, the video stream may include N frames of images, such as the 1st frame, the 2nd frame, ... the Nth frame. When displaying the video stream, the electronic device may display the 1st frame, the 2nd frame, ... the Nth frame on the display screen respectively. An application that wants to play the video stream may issue rendering commands for different frame images to the electronic device. The electronic device may render each frame image according to these rendering commands, obtain the rendering results (such as image data) of each frame image, and display based on the rendering results.

[0036] With the iterative updates of applications, the image rendering process for electronic devices based on rendering commands has become increasingly complex, resulting in an increasing resource load on electronic devices. Furthermore, the battery life of electronic devices is limited, making it difficult to reduce unnecessary image rendering overhead while ensuring image quality. To address this, researchers have developed predictive frame technology, which generates a single image based on two frames. For distinguishability, the image used to generate the predicted frame is called the real image.

[0037] Figure 1 is a schematic diagram of generating a predicted frame. The process of generating a predicted frame is: the electronic device separates the user interface (UI) of the first real image and the second real image to obtain a first main image of the first real image and a first UI image of the first real image, a second main image of the second real image, and a second UI image of the second real image. The first real image is the previous frame image of the second real image, the UI image includes the UI interface in the real image, and the main image includes objects such as the environment, environmental shadows, moving objects, and shadows of moving objects in the real image. The electronic device calculates a motion vector (MV) based on the first main image and the second main image to obtain a motion vector, which indicates the moving direction and speed of each object in the second main image; performs image warping (ImageWarp) on the second main image based on the motion vector to obtain a first intermediate image, wherein ImageWarp uses the motion vector to move each object in the second main image; since the moving speed and direction of each object in the second main image are different, image data is missing in the first intermediate image, so after obtaining the first intermediate image, image completion (HoleFilling) is performed on the first intermediate image to obtain a second intermediate image; the second intermediate image is fused with the second UI image (that is, the second UI image is attached to the second intermediate image) to generate a predicted frame, thereby generating a predicted frame based on the first real image and the second real image.

[0038] In some examples, the predicted frame can be an intermediate frame between the first real image and the second real image; in some examples, the predicted frame can be the next frame of the second real image. In addition, after obtaining the motion vector, the electronic device can perform ImageWarp on the first main image based on the motion vector to obtain a first intermediate image. Correspondingly, after obtaining the second intermediate image, the electronic device can fuse the second intermediate image with the first UI image to generate the predicted frame.

[0039] In order to enhance the user's sense of immersion, the electronic device can add shadow rendering to the image rendering, that is, for the moving object in the first real image and the second real image, the electronic device renders a shadow for the moving object, and the shadow moves with the movement of the moving object. For example, for the characters in the game application, the shadow is rendered for the character, and when the character moves, the shadow of the character moves with the character. So when the predicted frame of the character is generated, the predicted frame also has a shadow. However, in addition to the moving object and the shadow of the moving object, a frame of image also includes the environment. The environment refers to the scene where the moving object is located. The electronic device also renders a shadow for the environment, which is called an environmental shadow. When the electronic device generates a predicted frame, the shadow of the moving object moves according to the moving direction and speed of the environment. There is a phenomenon of incorrectly moving the shadow of the moving object, resulting in shadow jitter in the predicted frame during the movement of the moving object, increasing the difference between the predicted frame and the real image, and greatly reducing the user experience.

[0040] To address this issue, the present application provides an image processing method that, during the generation of a predicted frame, reuses image data of the shadow of a moving object in a real image into the predicted frame to generate the shadow of the moving object in the predicted frame. The moving object, environment, and environmental shadow in the predicted frame are generated using motion vectors, a main image of a real image frame, and a UI image of the real image. For example, the image data of the shadow of a moving object in the first real image or the second real image is reused.

[0041] To reuse image data of the shadow of a moving object in a real image, the shadow of the moving object generated based on the motion vector is removed during the generation of the predicted frame. Furthermore, when rendering the real image, the electronic device obtains image data of the shadow of the moving object in the real image and the position of the shadow of the moving object in the real image. The position of the shadow of the moving object in the real image is used to determine the position of the shadow of the moving object in the predicted frame, and the image data of the shadow of the moving object in the real image is reused in the predicted frame. As a result, the shadow of the moving object in the predicted frame no longer depends on the direction and speed of movement of the environment. Instead, it is generated by reusing the image data of the shadow of the moving object in the real image in the predicted frame. This reduces the shadow jitter problem in the predicted frame, reduces the difference between the predicted frame and the real image, and improves the user experience. The main image of the real image includes the shadow of the moving object. Reusing the image data of the shadow of the moving object in the real image can refer to reusing the image data of the shadow of the moving object in the main image of the real image. The position of the shadow of the moving object in the real image can refer to the position of the shadow of the moving object in the main image. The image data of the shadow of the moving object in the reusing main image can also be referred to as the image data of the shadow of the moving object in the main image.

[0042] The application's shadow rendering logic is as follows: when the shadow of a moving object is located in the environmental shadow, the moving object's shadow is not rendered in the real image. Therefore, when determining the position of the moving object's shadow in the real image, the shadow that is located in both the moving object's shadow and the environmental shadow is removed. Therefore, in some examples, the image data of the moving object's shadow in the main image is image data that is located in the moving object's shadow but not in the environmental shadow, that is, partial image data of the moving object's shadow in the main image. For example, in step S306 below, the entire image data of the moving object's shadow is obtained, and the image data of the moving object's shadow in the main image is partial image data of the entire image data obtained in step S306. In some examples, the image data of the moving object's shadow in the main image is the same as the entire image data obtained in step S306.

[0043] The solution provided in the embodiments of the present application is described in detail below with reference to the accompanying drawings.

[0044] It should be noted that the image processing method provided in the embodiment of the present application can be applied to the user's electronic device. The electronic device can be a device that can provide network access. For example, the electronic device can be a mobile phone, tablet computer, desktop, laptop, notebook computer, ultra-mobile personal computer (Ultra-mobile Personal Computer, UMPC), handheld computer, netbook, personal digital assistant (Personal Digital Assistant, PDA), wearable electronic device, smart watch and other devices. The present application does not specifically limit the specific form of the electronic device. In some embodiments, the electronic device can have a display function. For example, the electronic device can generate a predicted frame based on the first real image and the second real image, display the predicted frame, and display the first real image and the second real image.

[0045] As shown in Figure 2, the electronic device may include: a processor, an external memory interface, an internal memory, a Universal Serial Bus (USB) interface, a charging management module, a power management module, a battery, antenna 1, antenna 2, a mobile communication module, a wireless communication module, a sensor module, buttons, a motor, an indicator, a camera, a display, and a Subscriber Identity Module (SIM) card interface. The audio module may include a speaker, a receiver, a microphone, and a headphone jack, and the sensor module may include a pressure sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, and a bone conduction sensor.

[0046] It is understood that the structures illustrated in the embodiments of the present application do not constitute specific limitations on the electronic device. In other embodiments, the electronic device may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0047] The processor may include one or more processing units, for example, the processor may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processor (NPU). Different processing units may be independent devices or integrated into one or more processors. The processor is the nerve center and command center of the electronic device. The controller can generate operation control signals based on instruction opcodes and timing signals to complete the control of instruction fetching and execution.

[0048] The external memory interface can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device. The external memory card communicates with the processor through the external memory interface to implement the data storage function. The internal memory can be used to store computer executable program code, which includes instructions. The processor executes various functional applications and data processing of the electronic device by running the instructions stored in the internal memory. For example, in an embodiment of the present application, the processor causes the electronic device to execute the image processing method provided in an embodiment of the present application by running the instructions stored in the internal memory.

[0049] Electronic devices implement display functions through GPUs, displays, and application processors. A GPU is a microprocessor for image processing that connects the display and application processor. The GPU performs mathematical and geometric calculations for image rendering. A processor may include one or more GPUs, which execute program instructions to generate or modify display information.

[0050] The display screen is used to display images, videos, and the like. The display screen includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-OLED, or a quantum dot light-emitting diode (QLED). In some embodiments, the electronic device can include one or N display screens, where N is a positive integer greater than one.

[0051] A series of graphical user interfaces (GUIs) can be displayed on the display screen of an electronic device, and these GUIs are the main screen of the electronic device. Generally speaking, the size of the display screen of an electronic device is fixed, and only limited controls can be displayed on the display screen of the electronic device. A control is a GUI element, which is a software component included in an application that controls all data processed by the application and interactive operations on this data. Users can interact with the control through direct manipulation to read or edit relevant information of the application. Generally speaking, controls can include visual interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, etc. For example, in an embodiment of the present application, the display screen can display a video file containing specific actions, etc.

[0052] In some embodiments, the processor includes a CPU and a GPU. The CPU can generate a rendering instruction stream based on the rendering logic of the application and send the rendering instruction stream to the GPU so that the GPU can perform corresponding rendering according to the rendering instruction stream. The CPU can send rendering instructions of the rendering instruction stream to the GPU one by one. In some examples, the CPU can send rendering instructions to the GPU again after receiving a response from the GPU; in some examples, the CPU does not need to wait for the GPU's response and continues to send the next rendering instruction after completing the sending of one rendering instruction. The specific settings of the rendering instructions by the CPU are determined.

[0053] In an embodiment of the present application, when the GPU is rendering a real image, it can obtain image data of the shadow of a moving object in the real image, so as to reuse this image data when generating a predicted frame. As shown in Figure 3, it shows the process of generating a real image in the image processing method provided by an embodiment of the present application. In the process of generating a real image, obtaining image data of the shadow of a moving object in the real image can include the following steps:

[0054] S101: Receive a rendering instruction stream. The rendering instruction stream includes at least one rendering instruction, which instructs the GPU to perform corresponding rendering, such as instructing the GPU to render a moving object in a real image, instructing the GPU to render an environment in a real image, etc.

[0055] S102 : Determine whether to enable shadow rendering according to the rendering instruction stream. If yes, execute step S103 ; if not, execute step S107 .

[0056] In an embodiment of the present application, an offline rendering analysis is performed on the application in advance. The calling patterns and characteristics of the rendering instructions when the application turns shadow rendering on and off are compared to obtain the judgment conditions used by the application to identify whether shadow rendering is turned on. After receiving the rendering instruction stream, it is determined whether the information carried by the rendering instruction stream meets the judgment conditions. If so, shadow rendering is determined to be turned on; if not, shadow rendering is determined to be turned off.

[0057] For example, in some examples, when an application turns on shadow rendering, the application calls at least one of a specific program, a specific resource, and a specific rendering instruction. Thus, the judgment condition can record the information called when the application turns on shadow rendering. If the application calls a specific program when turning on shadow rendering, the judgment condition can record the label of the specific program to identify the specific program by the label, for example, the label of the specific program can be the program name of the specific program; if the application calls a specific resource when turning on shadow rendering, the judgment condition can record the resource label of the specific resource to identify the specific resource by the resource label, for example, the resource label can be the resource name, etc.; if the application calls a specific instruction when turning on shadow rendering, the judgment condition can record the specific instruction.

[0058] For example, when a game application turns on shadow rendering, it calls glClearColor(1.0,1.0,1.0,0.0). Therefore, the judgment condition is glClearColor(1.0,1.0,1.0,0.0), which requires calling glClearColor(R,G,B,A) and requires (RGBA == 1.0,1.0,1.0,0.0). Therefore, when the rendering instruction -glClearColor(1.0,1.0,1.0,0.0) exists in the rendering instruction stream, it is determined that shadow rendering is turned on, where R, G, B, and A are red, green, blue, and transparency, respectively.

[0059] S103: Render ambient shadows according to the rendering instructions related to ambient shadow rendering (ScreenSpaceShadowAmbientShadow) in the rendering instruction stream. Rendering ambient shadows is to generate image data of ambient shadows.

[0060] S104 : intercepting the rendering of the ambient shadow when rendering the ambient shadow to obtain image data of the ambient shadow.

[0061] S105 , according to the rendering instruction (ScreenSpaceShadow Human Shadow) related to the shadow rendering of the moving object in the rendering instruction stream, render the shadow of the moving object (Human Shadow). Rendering the shadow of the moving object is to generate image data of the shadow of the moving object.

[0062] S106 : intercepting the rendering of the shadow of the moving object when rendering the shadow of the moving object, so as to obtain image data of the shadow of the moving object.

[0063] S107: Execute corresponding rendering according to the rendering instruction stream to obtain all image data of the real image. Steps S103 to S106 complete the rendering of environmental shadows and shadows of moving objects. A frame of real image may also include moving objects, the environment, and a user interface. Therefore, in some examples, the rendering here primarily refers to rendering of the image other than environmental shadows and shadows of moving objects, such as rendering of moving objects, the environment, and a user interface. In some examples, the rendering of moving objects and the environment may be performed before shadow rendering. In this case, the rendering in step S107 may be rendering of the image other than environmental shadows, shadows of moving objects, moving objects, and the environment, such as rendering of a user interface.

[0064] In some examples, during the process of executing step S104 to obtain image data of the environment shadow, step S105 can be executed to render the shadow of the moving object. In some examples, during the process of executing step S106 to obtain image data of the shadow of the moving object, step S107 can be executed to perform corresponding rendering according to the rendering instruction stream to complete the rendering of the real image.

[0065] S108: Output the real image based on all the image data of the real image. After obtaining all the image data of the real image, generate the real image based on all the image data of the real image and display the real image.

[0066] In the process shown in FIG3 , image data of environmental shadows and image data of shadows of moving objects are obtained during the process of generating a real image. FIG4 shows a process of generating a prediction frame, in which image data of environmental shadows and image data of shadows of moving objects are used, and the process may include the following steps:

[0067] S201: Input a first real image and a second real image.

[0068] S202: Perform MV calculation based on the first main image of the first real image and the second main image of the second real image to obtain motion vectors. That is, perform MV calculation on the first main image and the second main image to obtain motion vectors, which indicate the movement direction and speed of each object in the second main image.

[0069] S203: Determine whether to enable shadow rendering. If yes, execute step S204; if no, execute step S208. The determination of whether to enable shadow rendering can refer to the above step S102 and will not be repeated here.

[0070] S204 : Determine the region where the shadow of the moving object is located in the real image according to the image data of the environmental shadow and the image data of the shadow of the moving object.

[0071] S205 : Perform image distortion on the second main image according to the motion vector to obtain a first intermediate image, and remove the shadow of the moving object in the first intermediate image.

[0072] The image warping of the second main image based on the motion vector primarily utilizes the motion vector to move each object in the second main image. Specifically, any object in the second main image is moved based on the object's movement direction and speed as determined by the motion vector. The second main image includes the environment, its shadow, the moving object, and its shadow. When moving each object in the second main image based on the motion vector, the moving object and the environment are first separated from the second main image. This separation separates the environment and the moving object's shadow from the moving object, effectively splitting the second main image into two parts: a first part including the environment, its shadow, and the moving object's shadow; and a second part including the moving object.

[0073] Then, the objects in the first portion are moved using the movement direction and speed of each object in the first portion of the motion vector; the moving objects are then moved using the movement direction and speed of the moving objects in the second portion of the motion vector. In some examples, when moving the shadow of the moving object, reuse of the image data of the moving object's shadow in the real image is prohibited. This method allows the shadow of the moving object to be removed from the first intermediate image, and the shadow removal of the moving object can be completed during the generation of the first intermediate image. In some examples, the shadow of the moving object in the first intermediate image is removed after the first intermediate image is generated. Regardless of the method, the first intermediate image obtained in step S205 is an image that contains the environment, the environment's shadows, and the moving object, but without the shadow of the moving object.

[0074] The purpose of removing the moving object's shadow is to mark the location of the moving object's shadow in the first intermediate image. Image data for the moving object's shadow in the real image is not used in generating the first intermediate image, thereby reserving a pixel-to-be-completed area at the marked location. The location of the moving object's shadow marked in the first intermediate image is obtained by shifting the moving object's shadow in the real image according to the motion vector. Therefore, the location of the moving object's shadow marked in the first intermediate image differs from its location in the real image.

[0075] S206 . Multiplex the image data of the shadow of the moving object in the second main image into the first intermediate image.

[0076] In some examples, multiplexing the image data of the moving object's shadow in the second main image involves not only multiplexing the image data but also multiplexing the position of the moving object's shadow in the second main image, so that the moving object's shadow in the second main image is superimposed on the first intermediate image. However, the position of the moving object's shadow marked in the first intermediate image differs from its position in the second main image. Therefore, after multiplexing the image data of the moving object's shadow, there is still an uncompleted area at the position of the moving object's shadow marked in the first intermediate image, where pixel values ​​need to be completed. The pixel values ​​of each pixel in this uncompleted area are obtained through image completion processing.

[0077] S207 : Perform image distortion on the second main image according to the motion vector to obtain a first intermediate image.

[0078] S208 . Perform hole filling processing on the first intermediate image to obtain a second intermediate image.

[0079] Because the first intermediate image is generated by moving objects in the second main image, the objects in the second main image may move at different speeds and directions, resulting in missing image data in the first intermediate image. This area with missing image data is called the "to-be-completed area." The pixel values ​​of each pixel in the "to-be-completed area" are obtained through image completion processing.

[0080] In one implementation, a deep learning network can be used to complete the pixel values ​​of each pixel in the area to be completed. In another implementation, a traditional image restoration operation can be used to complete the pixel values ​​of each pixel in the area to be completed. In another implementation, a mask image of the first intermediate image is used to complete the pixel values ​​of each pixel in the area to be completed, and the pixel values ​​of these static pixels are used to determine the pixel values ​​of the pixels to be completed.

[0081] The mask image includes three types of pixels: pixels to be complemented, static pixels, and dynamic pixels. The pixels to be complemented refer to the pixels in the to-be-complemented area of ​​the mask image (the same as the first intermediate image) whose pixel values ​​need to be complemented. Generally, a moving object and its shadow correspond to a to-be-complemented area, which can be a blank area. The pixels in the blank area near the moving object are the pixels to be complemented, and the pixels in the blank area near the moving object's shadow are the pixels to be complemented. Static pixels refer to pixels corresponding to a stationary environment in the mask image, whose pixel values ​​can be used to determine the pixel values ​​of the pixels to be complemented. Dynamic pixels refer to pixels corresponding to objects in motion (or called moving objects) in the mask image, whose pixel values ​​cannot be used to determine the pixel values ​​of the pixels to be complemented.

[0082] That is, the static pixel point is a valid reference point for completing the pixel point to be completed, and the dynamic pixel point is an invalid reference point for completing the pixel point to be completed. In the mask image, the above three types of pixel points can be distinguished by grayscale values, and the grayscale values ​​of the three types of pixel points are different. The grayscale value of the pixel point to be completed is a first grayscale value, the grayscale value of the static pixel point is a second grayscale value, and the grayscale value of the dynamic pixel point is a third grayscale value. Among them, the first grayscale value, the second grayscale value and the third grayscale value are not equal to each other. For example, the grayscale value of the static pixel point is 0, the grayscale value of the dynamic pixel point is 1, and the grayscale value of the pixel point to be completed is 0.5.

[0083] For each pixel to be filled in the mask image, the electronic device determines a set of valid static pixels corresponding to the pixel to be filled in the mask image and determines the pixel value of the pixel to be filled based on the pixel values ​​of this set of valid static pixels in the first intermediate image. One possible implementation is to take a weighted average of the pixel values ​​of the set of valid static pixels in the first intermediate image as the pixel value of the pixel to be filled. The electronic device then fills the first intermediate image with pixel values ​​based on the pixel values ​​of each pixel to be filled, generating a second intermediate image.

[0084] The effective static pixel points corresponding to a pixel to be filled have the same pixel spacing as the pixel to be filled, and the effective static pixel points have the smallest pixel spacing with the pixel to be filled compared to other static pixels (which may be called invalid static pixels).

[0085] S209: Post-process the second intermediate image. In some examples, the post-processing mainly involves performing image filtering on the second intermediate image.

[0086] S210 : Fusing the second UI image of the second real image and the processed second intermediate image to obtain a predicted frame.

[0087] It can be seen from Figures 3 and 4 above that in the process of generating a real image, image data of environmental shadows and image data of shadows of moving objects are obtained; in the process of generating a predicted frame, the image data of environmental shadows and image data of shadows of moving objects are used to determine the position of the shadow of the moving object in the real image, and the position of the shadow of the moving object in the real image is used to remove the shadow of the moving object generated according to the motion vector to mark the position of the shadow of the moving object generated according to the motion vector in the image, and then the image data of the shadow of the moving object in the real image is reused to generate the shadow of the moving object in the predicted frame. As a result, the shadow of the moving object in the predicted frame can be extracted from the environment of the predicted frame, so that the shadow of the moving object in the predicted frame no longer depends on the motion vector generation, but is generated by reusing the image data of the shadow of the moving object in the real image in the predicted frame, thereby reducing the shadow jitter problem existing in the predicted frame, reducing the difference between the predicted frame and the real image, and improving the user experience.

[0088] Furthermore, in scenarios where predictive frames suffer from shadow jitter, the image processing methods illustrated in Figures 3 and 4 involve four key components: 1. Acquiring image data of environmental shadows and shadows of moving objects in the real image; 2. Determining the position of the moving object's shadow in the real image; 3. Subtracting the moving object's shadow generated based on the motion vector; and 4. Reusing the image data of the moving object's shadow in the real image. These four components are explained below with reference to the accompanying figures.

[0089] FIG5 shows a process for obtaining image data of environmental shadows and image data of shadows of moving objects in a real image, which may include the following steps:

[0090] S301: Determine whether to enable shadow rendering and determine that the frame buffer (FB) bound to the current application is the first FB. The first FB may be GameFB, which can serve as the default FB for the application. Image data obtained by performing corresponding rendering according to the rendering instruction stream may be stored in GameFB. Determining whether to enable shadow rendering can be described in step S102 above and will not be repeated here.

[0091] S302: Render the environment shadow to generate image data of the environment shadow. The application's rendering instruction stream instructs that the environment shadow be rendered first, and then the shadow of the moving object be superimposed on the environment shadow to prevent the environment shadow from overlapping the shadow of the moving object. Therefore, in this embodiment, the environment shadow is rendered first, and the image data of the environment shadow is stored in the first FB.

[0092] S303: Create a second FB and a third FB. The second FB may be an AmbientFB, and the third FB may be a HumanFB. The purpose of creating the second FB and the third FB is to use the two FBs to store image data of environmental shadows and image data of shadows of moving objects, respectively, during the process of generating a real image, so as to intercept the image data of environmental shadows and image data of shadows of moving objects during the process of generating a real image.

[0093] In some examples, step S303 is executed after determining to turn on shadow rendering, that is, step S303 and step S302 can be executed in parallel; in some examples, step S303 can be executed before step S302, which is not limited here.

[0094] S304: Copy the image data of the ambient shadow to the second frame buffer, implementing independent storage of the image data of the ambient shadow using the second frame buffer, thereby intercepting the image data of the ambient shadow without hindering the generation of the real image. In some examples, the glBlitFramebuffer() function is called to copy the image data of the ambient shadow to the second frame buffer.

[0095] S305: Change the FB bound to the application from the first FB to the third FB. One possible implementation is to call the glBindFrameBuffer(Gl_DRAW_BUFFER, ID) function, where ID is the ID of the third FB, thereby changing the FB bound to the application from the first FB to the third FB.

[0096] After completing the rendering of the environmental shadow, you can render the shadow of the moving object. If the FB bound to the application is the first FB when rendering the shadow of the moving object, the image data of the shadow of the moving object will be stored in the first FB. The image data of the shadow of the moving object and the image data of the environmental shadow will be mixed and stored in the first FB, making it difficult to obtain the image data of the shadow of the moving object from the first FB. Therefore, before starting to render the shadow of the moving object, the FB bound to the application is changed from the first FB to the third FB. The image data of the shadow of the moving object generated when rendering the shadow of the moving object is stored in the third FB, thereby intercepting the image data of the shadow of the moving object in the process of generating the real image.

[0097] S306: Render the shadow of the moving object to generate image data of the shadow of the moving object.

[0098] S307: Change the FB bound to the application from the third FB to the first FB, i.e., restore the default FB of the application. For example, by calling the glBindFrameBuffer(Gl_DRAW_BUFFER, ID) function, where ID is the ID of the first FB, the FB bound to the application is changed from the third FB to the first FB.

[0099] S308. Render the shadow of the moving object again to generate image data of the shadow of the moving object. Because the FB bound to the application is changed to the first FB, the image data of the shadow of the moving object is stored in the first FB, ensuring that the rendering result of the application remains unchanged, that is, ensuring that the image data of the environment shadow and the image data of the shadow of the moving object are stored in the first FB of the application by default, and the image data of these two parts of the shadow can be read from the first FB during the process of generating the real image, preventing the real image from lacking shadows. In some examples, after the image data of the shadow of the moving object is generated, it can be copied to the first FB, which can also ensure that the image data of the environment shadow and the image data of the shadow of the moving object are stored in the first FB, but the efficiency of copying the image data is lower than the efficiency of rendering the shadow of the moving object.

[0100] In some examples, the first to third FBs may store pixel values ​​of an R channel in the image data to reduce occupancy of the first to third FBs.

[0101] As can be seen from the process shown in FIG. 5 , after generating the image data for the ambient shadow, the image data is copied to the second FB, so that both the first FB and the second FB store the image data for the ambient shadow. Furthermore, before generating the image data for the moving object's shadow, the third FB is bound to the application. This allows the image data for the moving object's shadow to be stored in the third FB when rendering the moving object's shadow. This allows the image data for the ambient shadow to be stored in the second FB and the image data for the moving object's shadow to be stored in the third FB, thereby intercepting the image data for the ambient shadow and the image data for the moving object's shadow during the real image generation process. Furthermore, the image data for the ambient shadow and the image data for the moving object's shadow are stored in different FBs, preventing the two pieces of image data from overlapping and becoming indistinguishable. While intercepting the image data for the ambient shadow and the image data for the moving object's shadow, the image data for the ambient shadow and the image data for the moving object's shadow can be stored in the application's default first FB, ensuring the accuracy of the image data stored in the first FB and preventing the real image from lacking shadows.

[0102] The application's shadow rendering logic is as follows: when the shadow of a moving object is within the ambient shadow, the shadow of the moving object is not rendered in the real image. Therefore, the image data of the ambient shadow and the image data of the moving object's shadow are used to determine whether the shadow of the moving object is within the ambient shadow, thereby determining the position of the shadow of the moving object in the real image. As shown in Figure 6, it shows the process of determining the position of the shadow of the moving object in the real image, which can include the following steps:

[0103] S401 : performing pixel-by-pixel comparison calculation on the image data of the environment shadow and the image data of the shadow of the moving object.

[0104] In this embodiment, the image data of the ambient shadow can generate an ambient shadow texture image, and the image data of the moving object's shadow can generate a moving object shadow texture image. The pixel-by-pixel comparison calculation can be performed on pixels with the same coordinates in the ambient shadow texture image and the moving object shadow texture image.

[0105] S402: For each pixel, determine whether the pixel is located in the shadow of the moving object. If yes, execute step S403; if not, execute step S405.

[0106] S403: Determine whether the pixel point is located in the ambient shadow. If not, execute step S404; if yes, execute step S405.

[0107] S404: Mark the pixel point as being in the shadow area of ​​the moving object.

[0108] S405: Mark that the pixel point is not in the shadow area of ​​the moving object.

[0109] When a pixel is located in the shadow of a moving object but not in the shadow of the environment, it means that the pixel is a pixel in the shadow of the moving object, and the pixel is marked as being in the shadow area of ​​the moving object. When a pixel is not located in the shadow of a moving object, it means that the pixel is not a pixel in the shadow of the moving object, and the pixel is marked as not being in the shadow area of ​​the moving object. When a pixel is located in the shadow of a moving object and also in the shadow of the environment, it means that the pixel is both a pixel in the shadow of the moving object and a pixel in the shadow of the environment. According to the principle that "when the shadow of a moving object is in the shadow of the environment, the shadow of the moving object is not presented in the real image", the pixel is marked as not being in the shadow area of ​​the moving object.

[0110] In one possible implementation, whether a pixel point is located in the shadow of a moving object or in the shadow of an environment is determined based on the pixel value of the pixel point. For example, hunman = texture(humanShadow, texCoords).r is used to obtain the pixel value of the pixel point with coordinates texCoords in the r channel (sometimes referred to as the R channel) from the moving object shadow texture image humanShadow; ambient = texture(ambientShadow, texCoords).r is used to obtain the pixel value of the pixel point with coordinates texCoords in the r channel from the environment shadow texture image ambientShadow. That is, through hunman = texture(humanShadow, texCoords).r and ambient = texture(ambientShadow, texCoords).r, the pixel value of the pixel point with the same coordinates in the r channel is obtained from the moving object shadow texture image humanShadow and the environment shadow texture image ambientShadow.

[0111] If ambient > hunman, it means that the pixel is in the shadow of the moving object, and the pixel is marked as being in the shadow area of ​​the moving object; if ambient ≤ hunman, it means that the pixel is in the environment shadow, and the pixel is marked as not being in the shadow area of ​​the moving object. Therefore, the two texture images are sampled through hunman = texture(humanShadow, texCoords).r and ambient = texture(ambientShadow, texCoords).r to obtain the pixel value of the r channel of each pixel with the same coordinates in the two texture images. Then, the pixel value is determined based on the size relationship of the pixel values ​​to determine whether it is in the shadow area of ​​the moving object or not.

[0112] After all pixels are compared, a mask image is generated, representing the shadow area of ​​the moving object in the real image. For example, when marking pixels, pixels in the shadow area of ​​the moving object are marked as 1, and pixels not in the shadow area of ​​the moving object are marked as 0. The 1 and 0 values ​​are used to determine the area of ​​the moving object's shadow in the real image, thereby determining the position of the moving object's shadow in the real image, that is, the position of the moving object's shadow in the main image of the real image.

[0113] FIG7 shows a process for removing the shadow of a moving object generated based on a motion vector. Taking removing the shadow of a moving object in a first intermediate image during the process of generating a first intermediate image based on a motion vector as an example, the process may include the following steps:

[0114] S501: Perform image warping on each pixel in the second main image based on a motion vector. The motion vector records the movement direction and speed of each pixel in the second main image. For each pixel in the second main image, image warping involves moving the pixel based on the direction and speed of movement. Each time a pixel is moved, a determination is made as to whether the pixel is within the shadow of the moving object, and a pixel value is determined based on the determination.

[0115] S502 , determining whether the pixel point is located in the shadow of the moving object according to the position of the shadow of the moving object in the second main image; if not, executing step S503 ; if yes, executing step S504 .

[0116] It can be understood that: each pixel point located in the shadow of the moving object in the second main image is still located in the shadow of the moving object after moving. Based on this principle, when image distortion is performed on the pixel point, it is determined whether the pixel point is located in the shadow of the moving object in the second main image. If so, it is determined that the pixel point is located in the shadow of the moving object; otherwise, it is determined that the pixel point is not located in the shadow of the moving object.

[0117] In other implementations, the position of the moving object's shadow in the second main image is known. Based on the movement direction and speed of the motion vector that matches the moving object's shadow, the position of the moving object's shadow in the first intermediate image can be determined in advance. After any pixel in the second main image is moved, the pixel's coordinates in the first intermediate image and the position of the moving object's shadow in the first intermediate image are used to determine whether the pixel is located in the moving object's shadow in the first intermediate image. For example, if the pixel's coordinates in the first intermediate image match the coordinates of a pixel in the moving object's shadow in the first intermediate image, the pixel is determined to be located in the moving object's shadow in the first intermediate image. Otherwise, the pixel is determined not to be located in the moving object's shadow in the first intermediate image.

[0118] For another example, a mask image is generated based on the position of the shadow of the moving object in the first intermediate image. The pixel points in the mask image that are in the shadow area of ​​the moving object are marked as 1, and the pixel points that are not in the shadow area of ​​the moving object are marked as 0. After completing the movement of any pixel point in the second main image, the pixel value of the pixel point at the coordinates of the pixel point in the first intermediate image is searched from the mask image. If the pixel value of the pixel point at the coordinates is 1, it is determined that it is located in the shadow of the moving object. Otherwise, it is determined that it is not located in the shadow of the moving object.

[0119] S503: Fill the pixel value of the pixel point with the pixel value of the pixel point in the second main image.

[0120] S504: Discard the pixel value of the pixel in the second main image. For example, the pixel value of the pixel is set to a preset value, such as (RGBA == 1.0, 1.0, 1.0, 0.25). The preset value is used to distinguish the shadow of the moving object in the first intermediate image from other objects in the first intermediate image. The preset value is used to remove the shadow of the moving object in the first intermediate image and mark the position of the shadow of the moving object in the first intermediate image.

[0121] S505: After completing the image warping for each pixel in the second primary image, a first intermediate image is generated from which the shadow of the moving object is removed, thereby obtaining a first intermediate image having the environment, the environmental shadow, and the moving object, but from which the shadow of the moving object is removed. The shadow of the moving object is removed by setting the pixel values ​​of each pixel in the shadow of the moving object in the first intermediate image to a preset value.

[0122] FIG8 is a process for multiplexing image data of shadows of moving objects in real images provided by the present application, which may include the following steps:

[0123] S601 , sequentially obtain each pixel point in a first intermediate image from which the shadow of a moving object is removed.

[0124] S602. For each pixel, determine whether the pixel is located in the shadow of the moving object according to the position of the shadow of the moving object in the real image. If yes, execute step S603; if not, execute step S604.

[0125] Here, whether the pixel point is located in the shadow of the moving object refers to whether the pixel point is located in the shadow of the moving object in the real image, specifically, the shadow of the moving object in the main image of the real image. In one possible implementation, the position of the shadow of the moving object in the real image can be represented by the coordinates of each pixel point located in the shadow of the moving object. If the coordinates of the pixel point in the first intermediate image from which the shadow of the moving object is removed are the same as the coordinates of a pixel point located in the shadow of the moving object, then the pixel point in the first intermediate image from which the shadow of the moving object is removed is determined to be located in the shadow of the moving object. Otherwise, the pixel point in the first intermediate image from which the shadow of the moving object is removed is determined not to be located in the shadow of the moving object.

[0126] In one possible implementation, the position of the shadow of the moving object in the real image is represented by the mask image obtained in FIG6 above, specifically by a pixel point with a pixel value of 1 in the mask image. Then, when determining whether a pixel point in the first intermediate image for removing the shadow of the moving object is located in the shadow of the moving object, the coordinates of the pixel point are used to find the pixel value of the pixel point at the coordinates in the mask image. If the pixel value of the pixel point at the coordinates is 1, it is determined to be located in the shadow of the moving object. Otherwise, it is determined that it is not located in the shadow of the moving object.

[0127] S603: Reuse the pixel value of the pixel in the real image. That is, fill the pixel value of the pixel with the pixel value of the pixel in the real image. Because the pixel with the reused pixel value is located in the shadow of the moving object, the reuse here is the image data of the shadow of the moving object in the real image.

[0128] S604: Keep the pixel value of the pixel unchanged.

[0129] S605 : Generate a first intermediate image by multiplexing the image data of the shadow of the moving object in the real image, thereby generating a first intermediate image having the shadow of the moving object.

[0130] As can be seen from Figure 8 above, by utilizing the first intermediate image with the shadow of the moving object removed, the position of the shadow of the moving object in the real image, and the image data of the shadow of the moving object in the real image, the image data of the shadow of the moving object in the real image can be fused into the first intermediate image, thereby simply merging the first intermediate image with the shadow of the moving object, realizing the reuse of the image data of the shadow of the moving object in the real image, avoiding the continued use of the motion vector used by the environment to generate the shadow of the moving object, reducing the shadow jitter problem in the predicted frame, reducing the difference between the predicted frame and the real image, and improving the user experience.

[0131] In this embodiment, there are still some pixel points whose pixel values ​​have not been assigned (i.e., the above-mentioned pixel points to be complemented) in the first intermediate image with the shadow of the moving object. After generating the first intermediate image with the shadow of the moving object, image completion processing is performed on the first intermediate image, and then post-processing and fusion are performed to generate a predicted frame.

[0132] Taking a game application as an example, (1) in FIG9 shows a real image used to generate a prediction frame. In the process of generating the real image, the position of the character's shadow in the real image is determined (such as the position circled by the irregular shape in (1) in FIG9). The environment of the real image is distorted using the motion vector to obtain the intermediate image shown in (2) in FIG9, which removes the character's shadow and the character. The character is then distorted to obtain the intermediate image shown in (3) in FIG9, which removes the character's shadow and the character's shadow.

[0133] The shadow of the intermediate image shown in (3) of FIG9 is rendered. During the shadow rendering process, the image data of the shadow of the person in the real image shown in (1) of FIG9 is directly copied to the intermediate image shown in (3) of FIG9, and the image shown in (4) of FIG9 is obtained. In this image, there are areas to be completed near the person and near the shadow of the person. Image completion processing is performed on each pixel point in the area to be completed, and the image shown in (5) of FIG9 is obtained. The shadow of the person in the image shown in (5) of FIG9 is obtained by reusing the shadow of the person shown in (1) of FIG9, so the shadow of the person in the two frames of images is the same, specifically, the image data of the shadow of the person in the two frames of images and the position in the image are the same.

[0134] In addition, the present application also provides an electronic device, including: one or more processors and a memory, the memory is used to store computer program code, the computer program code includes computer instructions, when one or more processors execute the computer instructions, the electronic device performs the above-mentioned image processing method.

[0135] The present application also provides a computer-readable storage medium for storing a computer program, which, when executed, is specifically used to implement the above-mentioned image processing method.

[0136] The present application also provides a computer program product comprising instructions, which, when executed on a computer or a processor, enables the computer or the processor to execute the above-mentioned image processing method.

Claims

1. An image processing method, characterized in that: Applied to electronic equipment, the method includes: Acquire a first main image from a first real image and acquire a second main image from a second real image, wherein the first real image and the second real image are images rendered by the electronic device; performing motion vector calculation on the first main image and the second main image to obtain a motion vector; warping an environment in a target main image and a moving object in the target main image according to the motion vector to generate an environment in a prediction frame and the moving object in the prediction frame, wherein the target main image is a main image of the first main image and the second main image; Image data of the shadow of the moving object in the target main image is multiplexed in the predicted frame to generate the shadow of the moving object in the predicted frame.

2. The method according to claim 1, characterized in that The multiplexing of the image data of the shadow of the moving object in the target main image in the predicted frame includes: For each pixel in the predicted frame, determining whether the pixel is located in the shadow of the moving object in the target main image according to the position of the shadow of the moving object in the target main image; When determining that the pixel point is located in the shadow of the moving object in the target main image, multiplexing the pixel value of the pixel point in the target main image, where the pixel value of the pixel point in the target main image is the image data of the pixel point in the target main image; When it is determined that the pixel point is not located in the shadow of the moving object of the target main image, the pixel value of the pixel point is kept unchanged.

3. The method according to claim 1, characterized in that The motion vector is used to record the moving speed and moving direction of each pixel in the target main image, and the step of distorting the environment in the target main image and the moving object in the target main image according to the motion vector to generate the environment in the predicted frame and the moving object in the predicted frame includes: Moving each pixel in the target main image according to the moving speed and moving direction of each pixel in the target main image; During the process of moving the pixel point, determining whether the pixel point is located in the shadow of the moving object in the predicted frame according to the position of the shadow of the moving object in the target main image; When determining that the pixel point is located in the shadow of the moving object in the prediction frame, setting the pixel value of the pixel point to a preset value; When it is determined that the pixel point is not located in the shadow of the moving object in the prediction frame, the pixel value of the pixel point is set to the pixel value of the pixel point in the target main image.

4. The method according to claim 3, characterized in that The preset value includes a preset transparency, and the preset transparency is different from the transparency of other pixels in the target main image.

5. The method according to any one of claims 2 to 4, characterized in that The process of determining the position of the shadow of the moving object in the target main image includes: Comparing the same-positioned pixels to be compared in an environment shadow texture image and a moving object shadow texture image, wherein the environment shadow texture image is generated based on image data of the environment shadow in the target main image, and the moving object shadow texture image is generated based on image data of the shadow of the moving object in the target main image; When it is determined that the pixel to be compared is in the shadow area of ​​the moving object but not in the shadow of the environment, marking the pixel to be compared as being in the shadow area of ​​the moving object; When it is determined that the pixel to be compared is in the shadow area of ​​the moving object and in the shadow of the environment, mark that the pixel to be compared is not in the shadow area of ​​the moving object; Based on the pixel points to be compared marked as being in the shadow area of ​​the moving object, the position of the shadow of the moving object in the target main image is determined, and the image data of the shadow of the moving object in the target main image includes the pixel values ​​of the pixel points to be compared located in the position in the target main image.

6. The method according to claim 5, characterized in that If the pixel value of the pixel to be compared in the moving object shadow texture image is greater than the pixel value of the pixel to be compared in the environment shadow texture image, it is determined that the pixel to be compared is in the shadow area of ​​the moving object but not in the environment shadow; If the pixel value of the pixel to be compared in the moving object shadow texture image is less than or equal to the pixel value of the pixel to be compared in the environment shadow texture image, it is determined that the pixel to be compared is in the shadow area of ​​the moving object and in the environment shadow.

7. The method according to claim 6, characterized in that The pixel value of the pixel to be compared includes at least one of an R channel value, a G channel value, and a B channel value of the pixel to be compared.

8. The method according to any one of claims 1 to 7, characterized in that Before acquiring the first main image from the first real image and acquiring the second main image from the second real image, the method further includes: When rendering the real image corresponding to the target main image, image data of the environmental shadow and image data of the shadow of the moving object are obtained, and the image data of the environmental shadow and the image data of the shadow of the moving object are used to determine the position of the shadow of the moving object in the target main image. The position of the shadow of the moving object in the target main image and the image data of the shadow of the moving object are used to obtain image data of the shadow of the moving object to be multiplexed in the target main image.

9. The method according to claim 8, characterized in that The obtaining of image data of the environment shadow and image data of the shadow of the moving object when rendering the real image corresponding to the target main image comprises: Rendering the environment shadow of the real image to generate image data of the environment shadow, and storing the image data of the environment shadow in a first frame buffer; Create the second and third frame buffers; Copying the image data of the ambient shadow stored in the first frame buffer to the second frame buffer; Changing the frame buffer bound to the real image from the first frame buffer to the third frame buffer; The shadow of the moving object in the real image is rendered to generate image data of the shadow of the moving object, and the image data of the shadow of the moving object is stored in the third frame buffer.

10. The method according to claim 9, characterized in that The step of obtaining the image data of the environment shadow and the image data of the shadow of the moving object when rendering the real image corresponding to the target main image further includes: After rendering the shadow of the moving object in the real image, changing the frame buffer bound to the real image from the third frame buffer to the first frame buffer; The shadow of the moving object in the real image is rendered again to generate image data of the shadow of the moving object, and the image data of the shadow of the moving object is stored in the first frame buffer.

11. The method according to claim 9 or 10, characterized in that The copying of the image data of the ambient shadow stored in the first frame buffer to the second frame buffer includes: calling a glBlitFramebuffer function to copy the image data of the ambient shadow stored in the first frame buffer to the second frame buffer.

12. The method according to any one of claims 9 to 11, characterized in that Changing the frame buffer bound to the real image from the first frame buffer to the third frame buffer includes: calling a glBindFrameBuffer function to change the frame buffer bound to the real image from the first frame buffer to the third frame buffer.

13. The method according to any one of claims 1 to 12, characterized in that The method further includes: after generating the shadow of the moving object in the predicted frame, performing image completion on the predicted frame.

14. An electronic device, characterized in that: include: one or more processors and memory; The memory is used to store computer program code, which includes computer instructions. When the one or more processors execute the computer instructions, the electronic device executes the image processing method according to any one of claims 1 to 13.

15. A computer-readable storage medium, characterized in that Used to store a computer program, which, when executed, is specifically used to implement the image processing method according to any one of claims 1 to 13.

Images