Image processing method and electronic device

Abstract

An image processing method and an electronic device. In the method, an electronic device, when detecting a scenario where frame interpolation needs to be performed, acquires first 3D data of a preset object in a first natively rendered frame, acquires a first 2D texture map of the first natively rendered frame, acquires second 3D data of the preset object in a second natively rendered frame, and acquires a second 2D texture map of the second natively rendered frame, wherein the second natively rendered frame is a frame following the first natively rendered frame; determines a first motion vector on the basis of the first 3D data and the second 3D data; determines a second motion vector on the basis of the first 2D texture map and the second 2D texture map; and generates a predicted frame on the basis of the first motion vector and the second motion vector. By fusing 3D data of a preset object, the method can improve the quality of generated predicted frames and reduce the likelihood of problems such as motion blur and ghosting in predicted frames.

Description

Image processing method and electronic device

[0001] Cross-reference to related applications

[0002] The present application claims priority to the Chinese Patent Application No. 202410532439.2, filed on April 29, 2024, and entitled "Image processing method and electronic device", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0003] The present application relates to the technical field of electronic devices, and in particular to an image processing method and an electronic device. BACKGROUND

[0004] In a scenario where an electronic device displays multiple frames of images, high frame rate and low power consumption help to improve the user's experience. For example, in a game scenario, picture quality, frame rate and power consumption are three important factors affecting the game experience. For another example, in an animation playing scenario, frame rate and power consumption also play a crucial role in the animation viewing experience.

[0005] Frame interpolation refers to calculating a predicted frame through frame information of two original frames of frames, so as to achieve the improvement of frame rate. Compared with the original frame rendering mode, the frame interpolation mode has smaller power consumption. However, some current frame interpolation modes are prone to cause problems such as trailing and ghosting of objects with large displacement in the original frames of frames, resulting in picture quality degradation. Therefore, how to improve the quality of the predicted frame generated in the frame interpolation scenario is worth studying. SUMMARY

[0006] The present application provides an image processing method and an electronic device, which provide a new frame interpolation mode to improve the quality of the predicted frame generated in the frame interpolation scenario.

[0007] In a first aspect, the present application provides an image processing method, which can be executed by an electronic device. The method comprises: detecting a preset event, wherein the preset event is used to indicate that an interpolation function is in an open state; in response to the preset event, obtaining first 3D data of a preset object in a first native rendering frame, obtaining a first 2D texture map of the first native rendering frame, obtaining second 3D data of the preset object in a second native rendering frame, and obtaining a second 2D texture map of the second native rendering frame, wherein the second native rendering frame is a neighboring next frame of the first native rendering frame; determining a first motion vector based on the first 3D data and the second 3D data; determining a second motion vector based on the first 2D texture map and the second 2D texture map; and generating a predicted frame according to the first motion vector and the second motion vector.

[0008] In the above method, in an interpolation scenario, the electronic device can improve the accuracy of the motion vector used to generate the predicted frame by fusing the 3D data of the preset object, thereby improving the quality of the generated predicted frame and reducing the probability of problems such as ghosting and virtual image in the predicted frame.

[0009] Optionally, the preset object is pre-labeled.

[0010] In a possible design, the method further comprises: displaying the first native rendering frame; displaying the predicted frame; and displaying the second native rendering frame. In another possible design, the method further comprises: displaying the first native rendering frame; displaying the second native rendering frame; and displaying the predicted frame.

[0011] In the above design, in an interpolation scenario, the electronic device can use an interpolation mode or an extrapolation mode to display the predicted frame in an interpolated form, thereby achieving the purpose of improving the frame rate with lower power consumption.

[0012] In a possible design, the determining of the first motion vector based on the first 3D data and the second 3D data can be implemented as follows: the first motion vector is determined based on the first 3D data and the second 3D data by using a multi-level hash algorithm. For example, the electronic device can first divide the 3D data of the image into different rendering parts according to types, and perform unique matching processing of the first level of hash; then divide the remaining 3D data that is not uniquely matched into different rendering instances, and perform unique matching processing of the second level of hash; and finally calculate and match the remaining 3D data that is not uniquely matched according to spatial distance, to implement unique matching processing of the third level of hash. The first motion vector can be understood as a 3D motion vector.

[0013] In the above design, the multi-level hash algorithm can improve the calculation efficiency of the first motion vector.

[0014] In a possible design of the method, the generating the predicted frame according to the first motion vector and the second motion vector can be implemented as follows: determining a first weight factor of the first motion vector and a second weight factor of the second motion vector; performing weighted fusion on the first motion vector and the second motion vector according to the first weight factor and the second weight factor to obtain a third motion vector; and generating the predicted frame according to the third motion vector. The first motion vector can be understood as a 3D motion vector, the second motion vector can be understood as a 2D motion vector, and the third motion vector can be understood as a motion vector obtained after weighted fusion.

[0015] In the above design, the accuracy of the obtained motion vector can be improved by calculating the third motion vector by using the weighted fusion, and thus the quality of the generated predicted frame can be improved.

[0016] In a possible design of the method, the detecting the preset event includes but is not limited to the following: detecting a first operation on a first control, where the first control is used to manage the frame insertion function, and the first operation is used to start the frame insertion function.

[0017] In the above design, the electronic device provides the first control for starting the frame insertion function through the user interface, so that different user requirements for the frame insertion scenario can be met, and the diversity of application scenarios is improved.

[0018] According to the above possible designs, the first 3D data or the second 3D data can include but is not limited to at least one of the following data: 3D geometric vertex data, 3D polygon mesh data, and 3D geometric edge data.

[0019] In the above design, the calculation accuracy of the motion vector can be improved by fusing at least one type of 3D data, and thus the quality of the generated predicted frame can be improved.

[0020] In a second aspect, the present application provides an electronic device, which includes a plurality of function modules; the plurality of function modules interact with each other to implement the method performed by the electronic device in the first aspect and possible designs thereof. The plurality of function modules can be implemented based on software, hardware, or a combination of software and hardware, and the plurality of function modules can be combined or divided based on specific implementation.

[0021] In a third aspect, the present application provides an electronic device, which includes at least one processor and at least one memory, and the at least one memory stores computer program instructions. When the electronic device is running, the at least one processor executes the method performed by the electronic device in the first aspect and possible designs thereof.

[0022] In a fourth aspect, the present application also provides a chip for reading a computer program stored in a memory, and executing the method performed by the electronic device in any of the above aspects and possible designs thereof.

[0023] In a fifth aspect, the present application also provides a chip system, which comprises a processing circuit and a storage medium, and the storage medium stores instructions, and the instructions are executed by the processing circuit to implement the method performed by the electronic device in any of the above aspects and possible designs thereof. The chip system can be composed of a chip, or can include a chip and other discrete devices.

[0024] In a sixth aspect, the present application also provides a computer readable storage medium, which stores a computer program, and when the computer program is executed by a computer, the computer is caused to execute the method performed by the electronic device in any of the above aspects and possible designs thereof.

[0025] In a seventh aspect, the present application also provides a computer program product comprising instructions, and when the computer program product is run on a computer, the computer is caused to execute the method performed by the electronic device in any of the above aspects and possible designs thereof.

[0026] The technical effects that can be achieved by the technical solutions of any of the above second to seventh aspects can be described with reference to the technical effects that can be achieved by the technical solutions of the first aspect, and the repeated parts will not be described herein. BRIEF DESCRIPTION OF DRAWINGS

[0027] FIG. 1 shows a possible hardware structure of an electronic device;

[0028] FIG. 2 shows a possible software system architecture diagram of an electronic device;

[0029] FIG. 3 is a schematic diagram of an application scenario to which the image processing method provided by the embodiments of the present application is applied;

[0030] FIG. 4 is a schematic diagram of a process of pre-labeling a preset object provided by the embodiments of the present application;

[0031] FIG. 5 is a schematic diagram of a process of an image processing method provided by the embodiments of the present application;

[0032] FIG. 6 is a schematic diagram of a process of calculating 3D_MV provided by the embodiments of the present application;

[0033] FIG. 7 is a timing diagram of image processing in an interpolation mode provided by the embodiments of the present application;

[0034] FIG. 8 is a timing diagram of image processing in an extrapolation mode provided by the embodiments of the present application;

[0035] FIG. 9 is another flowchart of an image processing method according to an embodiment of the present application;

[0036] FIG. 10 is a possible structural diagram of an image processing device according to an embodiment of the present application;

[0037] FIG. 11 is a possible structural diagram of an apparatus according to an embodiment of the present application. DETAILED DESCRIPTION

[0038] The embodiments of the present application will be described in detail below with reference to the accompanying drawings and embodiments.

[0039] Electronic devices with display functions such as mobile phones, tablets, smart screens, and smart devices are becoming increasingly popular, and with the continuous development of electronic devices, the functions of electronic devices are gradually improving and enhancing. In the scenario where electronic devices display multiple frames of images such as game scenes, animation playing scenes, etc., an interpolation method can be used to improve the frame rate and reduce power consumption. The image processing method provided by the embodiments of the present application can be applied to the display scenario of multiple frames of images obtained based on 3D scene modeling. For example, a game scene obtained based on 3D scene modeling, game characters and / or game scenes in the game are obtained based on 3D scene modeling. For another example, an animation playing scene obtained based on 3D scene modeling, animation characters or animation scenes in the animation are obtained based on 3D scene modeling.

[0040] For ease of understanding, the following first introduces technical terms that the method provided by the present application can involve:

[0041] (1) Interpolation refers to generating a predicted frame, i.e., a new frame of image, based on the adjacent previous and next two original rendering frames. Optionally, if the predicted frame is displayed between the two original rendering frames, it is called interpolation. Alternatively, if the predicted frame is displayed after the display of the next original rendering frame, it is called extrapolation.

[0042] Through interpolation processing, the display time of each frame of image can be shortened, which can also be understood as being able to improve the display frame rate of the electronic device for images, for example, the original frame rate is 60 hertz (Hz), and after interpolation processing, it can be improved to 120 Hz, etc. In this way, the display stability of the electronic device for multiple frames of images can be improved, and a higher frame rate of use experience can be provided to the user.

[0043] (2) Motion estimate and motion compensation (MEMC) is an interpolation technology.

[0044] In a possible implementation, motion estimation is performed based on 2D texture maps of two original frames, and then warp processing and hole filling are performed on the two original frames by using the motion vectors, so as to obtain a prediction frame to be added. The warp processing refers to performing various operations on an image by using a rotation and scaling matrix, and the operations may, for example, include but are not limited to translation, rotation around a point, scaling, and the like. During the warp processing, holes (or "holes") are formed, and the prediction frame after optimization can be obtained by hole filling (or "hole filling") processing, so that the completeness of the prediction frame can be improved. However, the motion vectors obtained based on the 2D texture maps have large errors, which may result in serious ghosting and virtual images in the prediction frame. The motion estimation, motion compensation, and motion vector (MV) can be referred to the description in the related art, and will not be described in detail herein.

[0045] Therefore, an image processing method is provided in the embodiments of the present application. In the method, by pre-labeling a preset object, 3D data of the preset object can be obtained in the process of adding a prediction frame, so that a motion vector with higher accuracy can be obtained, and a prediction frame with higher quality can be generated. In this way, in the process of frame interpolation, the error of the obtained motion vector can be reduced by combining 3D data for frame interpolation, the picture quality in the frame interpolation scene can be improved, and the probability of problems such as ghosting and virtual images in the prediction frame can be reduced.

[0046] The embodiments of the present application can be applied to electronic devices with display functions, such as mobile phones, PCs, tablet computers, wearable devices (for example, watches, bracelets, etc.), vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, notebook computers, ultra-mobile personal computers (UMPC), netbooks, personal digital assistants (PDA), smart home devices (for example, smart TVs, smart speakers, etc.), and the like. It can be understood that the embodiments of the present application do not make any limitation on the specific type of electronic devices.

[0047] The electronic devices to which the embodiments of the present application can be applied include, for example, but are not limited to, electronic devices running or other operating systems. The above portable electronic devices can also be other portable electronic devices, such as laptops with a touch-sensitive surface (for example, a touch panel).

[0048] FIG. 1 shows a schematic diagram of a hardware structure of a possible electronic device. The electronic device 100 includes radio frequency (RF) circuitry 110, a power supply 120, a processor 130, a memory 140, an input unit 150, a display unit 160, audio circuitry 170, a communication interface 180, and a wireless-fidelity (Wi-Fi) module 190, etc. Those skilled in the art can understand that the hardware structure of the electronic device 100 shown in FIG. 1 does not constitute a limitation on the electronic device 100, and the electronic device 100 provided by the embodiments of the present application can include more or fewer components than those shown, can combine two or more components, or can have a different component configuration. The various components shown in FIG. 1 can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application-specific integrated circuits.

[0049] The various constituent components of the electronic device 100 will be described in detail below in conjunction with FIG. 1:

[0050] The RF circuitry 110 can be used in the process of communication or conversation, receiving and sending data. In particular, the RF circuitry 110 sends the downlink data received from the base station to the processor 130 for processing, and sends the uplink data to be sent to the base station. Typically, the RF circuitry 110 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (LNA), a duplexer, etc.

[0051] In addition, the RF circuitry 110 can also communicate with other devices through a wireless communication network. The wireless communication can use any communication standard or protocol, including but not limited to global system for mobile communication (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), long term evolution (LTE), email, short messaging service (SMS), etc.

[0052] The electronic device 100 can also implement communication services and interact with other electronic devices, and therefore the electronic device 100 needs to have a data transmission function, i.e., the electronic device 100 needs to include a communication module. Although FIG. 1 shows the RF circuit 110, the Wi-Fi module 190, and the communication interface 180 as communication modules, it can be understood that at least one of the above components or other communication modules (such as a Bluetooth module) for implementing communication exist in the electronic device 100 to perform data transmission.

[0053] For example, when the electronic device 100 is a mobile phone, the electronic device 100 can include the RF circuit 110, and can also include the Wi-Fi module 190 or a Bluetooth module (not shown in FIG. 1); when the electronic device 100 is a computer, the electronic device 100 can include the communication interface 180, and can also include the Wi-Fi module 190 or a Bluetooth module (not shown in FIG. 1); when the electronic device 100 is a tablet computer, the electronic device 100 can include the Wi-Fi module 190 or a Bluetooth module (not shown in FIG. 1).

[0054] Wi-Fi technology belongs to short-range wireless transmission technology, and the electronic device 100 can connect to an access point (AP) through the Wi-Fi module 190 to access a data network. The Wi-Fi module 190 can be used for data reception and transmission in the communication process.

[0055] The electronic device 100 can be physically connected to other devices through the communication interface 180. Optionally, the communication interface 180 and the communication interface of the other device are connected through a cable to realize data transmission between the electronic device 100 and the other device.

[0056] The memory 140 can be used to store software programs and modules. The processor 130 executes various function applications and data processing of the electronic device 100 by running the software programs and modules stored in the memory 140. Optionally, the memory 140 can mainly include a program storage area and a data storage area. The program storage area can store an operating system (mainly including respective software programs or modules corresponding to a kernel layer, a system layer, an application program framework layer, and an application program layer).

[0057] In addition, the memory 140 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other volatile solid-state memory device. In an embodiment of the present application, the memory 140 can store computer program instructions and the like for implementing the image processing method provided by the embodiment of the present application, for example, store the acquired 3D data, and for example, store computer program instructions for how to acquire the 3D data of the preset object, computer program instructions for how to generate the predicted frame.

[0058] The input unit 150 can be used to receive the editing operation of various types of data objects such as digital or character information input by the user, and to generate key signal input related to the user settings and function control of the electronic device 100. Optionally, the input unit 150 can include a touch panel 151 and other input devices 152. The touch panel 151, also known as a touch screen, can collect the touch operation of the user on or near it (such as the operation of the user using a finger, a stylus, or any suitable object or accessory on or near the touch panel 151), and drive the corresponding connection device according to the pre-set program. In an embodiment of the present application, the touch panel 151 can collect the user operation of the user on the display panel 161, and the user operation can be, for example, setting the state of the "high frame rate" or "super frame" control in the game application to on, and the like.

[0059] Optionally, the other input devices 152 can include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, on-off keys, etc.), trackballs, mice, joysticks, and the like. For example, when the electronic device 100 is a notebook computer, the other input devices 152 can be a mouse and / or a keyboard.

[0060] The display unit 160 can be used to display information input by a user or information provided to the user and various menus of the electronic device 100. The display unit 160 is a display system of the electronic device 100, used to present an interface and realize human-computer interaction. The display unit 160 can include a display panel 161. Optionally, the display panel 161 can be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like. In an embodiment of the present application, the display unit 160 can be used to display a user interface for a user, such as a game interface, an animation interface, and the like after processing by the method provided in the embodiment of the present application. It can be understood that the game interface includes not only a native rendering frame but also a predicted frame obtained by using the method provided in the present application, and similarly, the animation interface includes not only a native rendering frame but also a predicted frame obtained by using the method provided in the present application.

[0061] The processor 130 is the center of controlling the electronic device 100, and connects various components by using various interfaces and lines, executes various functions of the electronic device 100 and processes data by running or executing software programs and / or modules stored in the memory 140 and calling data stored in the memory 140, thereby realizing various services based on the electronic device 100. In an embodiment of the present application, the processor 130 can be used to implement the image processing method provided in the embodiment of the present application, so as to improve the quality of the predicted frame obtained in the interpolation scene and reduce problems such as ghosting and trailing caused by high displacement of dynamic objects; and compared with the native rendering frame mode, the method can also reduce the power consumption of the electronic device.

[0062] The electronic device 100 further includes a power supply 120 (such as a battery) for supplying power to various components. Optionally, the power supply 120 can be logically connected to the processor 130 through a power management system, so as to realize functions such as charging, discharging, and power consumption management through the power management system.

[0063] As shown in FIG. 1, the electronic device 100 further includes an audio circuit 170, a microphone 171, and a speaker 172, which can provide an audio interface between a user and the electronic device 100. The audio circuit 170 can be used to convert audio data into a signal recognizable by the speaker 172 and transmit the signal to the speaker 172 for conversion into an audible signal output. The microphone 171 is used to collect external sound signals (such as human speech or other sounds, etc.) and convert the collected external sound signals into a signal recognizable by the audio circuit 170 for transmission to the audio circuit 170. The audio circuit 170 can also be used to convert the signal transmitted by the microphone 171 into audio data, and output the audio data to the RF circuit 110 for transmission to, for example, another electronic device, or to the memory 140 for subsequent further processing.

[0064] Although not shown in FIG. 1, the electronic device 100 can also include a camera, at least one sensor, etc., which will not be described here. The at least one sensor can include, but is not limited to, a pressure sensor, a barometric pressure sensor, an acceleration sensor, a distance sensor, a fingerprint sensor, a touch sensor, a temperature sensor, etc.

[0065] Embodiments of the present application relate to an operating system (OS) running on the electronic device 100. The software system of the electronic device 100 can adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservice architecture, or a cloud architecture. Embodiments of the present application take the operating system with a layered architecture as an example to illustrate the software system architecture of the electronic device 100.

[0066] FIG. 2 shows a possible software system architecture block diagram of an electronic device. As shown in FIG. 2, the software system architecture of the electronic device can be a layered architecture, for example, the software can be divided into several layers, each layer has a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, the operating system is divided into five layers, from top to bottom, the application layer, the application framework layer (FWK), the runtime and system library, the kernel layer, and the hardware layer.

[0067] The application program layer can include a series of application packages. As shown in FIG. 2, the application program layer can include a camera, settings, third-party applications, and the like. Among them, the third-party applications can include a wireless local area network (WLAN), music, calls, Bluetooth, and the like, and can further include videos, games, and the like. Among them, the game application can further include a game engine / game plug-in and the like, which can facilitate the efficient and convenient development of game applications by developers. The video application can further include a 3D rendering software and the like, which can facilitate the efficient and convenient production of animations by animators.

[0068] In the embodiments of the present application, the application program layer can be used to realize the presentation of a display interface. The display interface can include a predicted frame obtained based on the image processing method in the following embodiments, and can also include a native rendering frame. Optionally, when interpolation is performed in an interpolation mode, the predicted frame is displayed between two native rendering frames. Alternatively, when interpolation is performed in an extrapolation mode, the predicted frame is displayed after the display of the two native rendering frames.

[0069] In a possible implementation manner, the application program can be developed using the java language, and is completed by calling an application programming interface (API) provided by the application framework layer. Developers can interact with the bottom layer (such as a hardware layer, a kernel layer, and the like) of the operating system through the application framework layer to develop their own application programs. The application framework layer is mainly a series of services and management systems of the operating system.

[0070] The application framework layer provides an application programming interface and a programming framework for the application programs of the application program layer. The application framework layer includes some predefined functions. As shown in FIG. 2, the application framework layer can include an activity manager, a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like. In the embodiments of the present application, the application framework layer can further include a software development kit (SDK) of the interpolation service. The SDK of the interpolation service is used to encapsulate the capabilities of the interpolation service, to realize the management of the interpolation service; and through an open interface, the SDK of the interpolation service can support the calling of the application program layer.

[0071] The activity manager is used to manage the life cycle of each application program and provide a commonly used navigation back function, and provides an interactive interface for the windows of all programs.

[0072] The window manager is used to manage windows programs. The window manager can acquire the display screen size, determine whether there is a status bar, lock the screen, and capture the screen, etc.

[0073] The content provider is used to store and acquire data, and make the data accessible to the application program. The data can include video, image, audio, dialed and received phone, browsing history and bookmark, phone book, etc.

[0074] The view system includes visual controls, such as a control for displaying text, a control for displaying pictures, etc. The view system can be used to build an application program. A display interface can be composed of one or more views. For example, a display interface including a short message notification icon can include a view for displaying text and a view for displaying pictures.

[0075] The phone manager is used to provide the communication function of the electronic device. For example, the management of the call state (including call connection, call hang-up, etc.).

[0076] The resource manager provides various resources for the application program, such as localized strings, icons, pictures, layout files, video files, etc.

[0077] The notification manager makes the application program display notification information in the status bar, which can be used to convey a type of message, which can automatically disappear after a short stay without user interaction. For example, the notification manager is used to inform the completion of the download, message reminder, etc. The notification manager can also be a notification in the form of a chart or a scroll bar text appearing in the top status bar of the system, such as a notification of an application program running in the background, and can also be a notification in the form of a dialogue window appearing on the screen. For example, the text information is prompted in the status bar, a prompt sound is emitted, the electronic device vibrates, the indicator light flashes, etc.

[0078] The runtime includes the core library and the virtual machine. The runtime is responsible for the scheduling and management of the operating system.

[0079] The core library includes two parts: one part is the function function called by the java language, and the other part is the core library of the operating system. The application program layer and the application program framework layer run in the virtual machine. The virtual machine executes the java file of the application program layer and the application program framework layer into a binary file. The virtual machine is used to perform the management of the object life cycle, the management of the stack, the management of the thread, the management of the security and the exception, and the garbage collection, etc.

[0080] The system library can include a plurality of functional modules. For example, a surface manager, a media framework, a three-dimensional graphics processing library (e.g., OpenGL ES), a two-dimensional graphics engine (e.g., SGL), and the like. In embodiments of the present application, the system library can also include a 2D motion vector (MV) module, a 3D MV module, a fusion module, and a frame generation module. Among them,

[0081] The 2D MV module is configured to calculate a 2D MV of a predicted frame based on a 2D texture map.

[0082] The 3D MV module is configured to calculate a 3D MV of a preset object in the predicted frame based on 3D data of the preset object. The 3D data can be at least one of 3D geometric vertex data, 3D polygon mesh data, 3D geometric edge data, and the like.

[0083] The fusion module is configured to obtain a fusion MV of the predicted frame based on fusion of the 2D MV and the 3D MV.

[0084] The frame generation module is configured to generate the predicted frame based on motion compensation of the fusion MV.

[0085] The three-dimensional graphics processing library is configured to implement three-dimensional graphics drawing, image rendering, synthesis, and layer processing, and the like.

[0086] The two-dimensional graphics engine is a drawing engine for two-dimensional drawing.

[0087] In some embodiments, the three-dimensional graphics processing library can be used to draw a three-dimensional motion trajectory image, and the two-dimensional graphics engine can be used to draw a two-dimensional motion trajectory image.

[0088] The surface manager is configured to manage a display subsystem and provide fusion of two-dimensional and 3D layers for a plurality of applications.

[0089] The media framework supports playback and recording of a plurality of commonly used audio, video formats, and static image files, and the like. The media framework can support a plurality of audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, and the like.

[0090] The kernel layer is a layer between hardware and software. The kernel layer at least includes a display driver, and the like.

[0091] The hardware layer can include various sensors, such as a touch sensor, and the like. Components included in the hardware layer can also be referred to in FIG. 1, and will not be described again here.

[0092] Generally, the electronic device 100 can run multiple application programs simultaneously. More simply, one application program can correspond to one process, and more complexly, one application program can correspond to multiple processes. Each process has a process number (process ID).

[0093] It should be understood that "at least one" or similar expressions in the embodiments of the present application refer to any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can represent: a, b, c, a and b, a and c, b and c, or a, b and c, where a, b, c can be single or multiple. "Multiple" means two or more. "And / or" describes the association between the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent: A exists alone, A and B exist together, and B exists alone, where A, B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects.

[0094] In addition, it should be understood that in the description of the present application, the words "first", "second", etc. are only used for the purpose of distinguishing the description, and cannot be understood as indicating or implying relative importance, nor indicating or implying order.

[0095] It should be understood that the hardware structure of the electronic device can be as shown in FIG. 1, and the software system architecture can be as shown in FIG. 2, wherein the software program and / or module corresponding to the software system architecture in the electronic device can be stored in the memory 140, and the processor 130 can run the software program and application stored in the memory 140 to execute the flow of the image processing method provided in the embodiments of the present application.

[0096] In order to facilitate understanding of the image processing method provided by the present application, the implementation process of the method provided by the present application will be introduced below in combination with the contents shown in FIGS. 3 to 8.

[0097] The embodiments of the present application are applicable to a display scenario of improving frame rate. For example, FIG. 3 is a schematic diagram of an application scenario to which the image processing method provided by the embodiments of the present application is applicable. As shown in interface 301 in FIG. 3, a possible setting interface included in an electronic device, for example, a game setting interface or a video setting interface, is shown. The control 300 of "high frame rate" in the interface 301 is in an off state. In the interface 301, the electronic device can display the interface 302 in response to an operation on the control 300. The operation on the control 300 may, for example, include but is not limited to a click operation, a double-click operation, a long-press operation or a non-contact operation, and the specific type of the operation is not limited in the present application. As can be seen from the interface 302, compared with the interface 301, the control 300 of "high frame rate" is switched to an on state, which can also be understood as indicating that the frame insertion function is in an on state.

[0098] Referring to the scenario shown in FIG. 3, the electronic device detects and responds to the setting of the high frame rate function to the on state, and can generate a frame prediction command to implement the frame insertion processing by using the image processing method provided by the embodiments of the present application. For example, taking the interface 302 as an example of a game setting interface, the game application interface included in the game application is displayed after the frame insertion processing. In order to facilitate understanding, the differences between two different setting scenarios are explained below by means of Table 1:

[0099] Table 1

[0100] As can be seen from Table 1, after the high frame rate function is set to the on state, the image frames included in the game application interface displayed by the electronic device can include not only native rendering frames but also prediction frames between two native rendering frames. For example, the prediction frame 1 can be calculated based on the frame information of the native rendering frame 1 and the native rendering frame 2, and the prediction frame 2 can be calculated based on the frame information of the native rendering frame 2 and the native rendering frame 3. In this way, the electronic device can improve the frame rate during the display of the game application interface, and can improve the stability and clarity of the interface display. It can be understood that the native rendering frame 2 is the adjacent next frame of the native rendering frame 1, and the native rendering frame 3 is the adjacent next frame of the native rendering frame 2.

[0101] It should be noted that FIG. 3 shows a possible scenario, and the high frame rate function shown in FIG. 3 can be adaptively turned on in response to user demand. In other possible scenarios, the high frame rate function may, for example, also be turned on by default, and the display scenario of improving frame rate is not limited in the embodiments of the present application.

[0102] In addition, the frame insertion result shown in Table 1 is only one possible example, and more or less frame insertion may be included in other possible examples, for example, the original frame rate is 60 frames, and the high frame rate is 90 frames.

[0103] In the display scene of the plurality of images based on the 3D scene modeling, the 3D objects are generally included in the images, for example, the 3D objects included in the game scene can be game characters, and the 3D objects included in the animation playing scene can be animation characters.

[0104] In an optional embodiment, in the process of developing the display scene, the preset object is obtained by pre-labeling the 3D object, so that in the process of the in-between frame processing, the 3D data of the preset object can be obtained based on the pre-labeled preset object, and then the 3D_MV can be obtained based on the 3D data. For example, FIG. 4 is a flowchart of pre-labeling a preset object according to an embodiment of the present application.

[0105] Step 401: obtaining a labeling instruction for the preset object.

[0106] For example, in the game development process, the labeling instruction for the preset object can be determined by detecting and responding to the labeling operation of the game developer on the game character.

[0107] Step 402: adding a first attribute to the preset object in response to the labeling instruction; wherein the first attribute is used to realize the labeling function of the preset object.

[0108] Correspondingly, in the application running process, the electronic device can detect the first attribute, and then determine that the pre-labeled preset object exists in response to the first attribute.

[0109] Step 403: adding a member variable and a member function corresponding to the first attribute.

[0110] The member variable and the member function are used to indicate that the 3D data of the preset object is obtained in the in-between frame process. Correspondingly, in the application running process, the electronic device can obtain the 3D data of the preset object according to the member variable and the member function corresponding to the first attribute.

[0111] In addition, the first attribute added through the process shown in FIG. 4 can be integrated in the application plug-in, for example, in the game plug-in. In this way, in the application running process, the 3D data can be obtained more accurately and efficiently based on the preset object by calling the function of the integrated first attribute in the application plug-in. The labeling method has the characteristics of small development amount.

[0112] Optionally, the preset object is determined based on the flow shown in FIG. 4, and a marking command for the preset object in the frame interpolation scene can be implemented in a driver development kit (DDK) layer. The marking command is generated by the electronic device detecting and responding to the first attribute. The DDK layer can be implemented in a system library, and is used to implement marking processing of the preset object by an operating system included in the electronic device, to obtain 3D data corresponding to the preset object.

[0113] It should be noted that the preset object can also be pre-marked by other marking manners in the embodiments of the present application, and the marking manner is not limited in the present application.

[0114] Referring to the application scenario introduced in FIG. 3, FIG. 5 is a flowchart of an image processing method provided by an embodiment of the present application. The flowchart can include the following steps:

[0115] In step 501, the electronic device detects and responds to a preset event to perform frame interpolation preprocessing. The preset event is used to indicate that the frame interpolation function is in an open state. For example, the electronic device detects the preset event, which can be implemented by detecting a first operation on a first control. The first control is used to manage the frame interpolation function. For example, the first control is the high frame rate control 300 shown in FIG. 3. When the high frame rate control 300 is displayed as open, it indicates that the frame interpolation function is in an open state. When the high frame rate control 300 is displayed as closed, it indicates that the frame interpolation function is in a closed state. It can be understood that the first operation is used to open the frame interpolation function. For example, the first operation is an operation of setting the high frame rate control to open in FIG. 3.

[0116] For example, the first application can be a game application, a video application, etc.

[0117] For example, the frame interpolation preprocessing can include but is not limited to: initializing an interface, initializing a frame interpolation instance, requesting CPU resources, GPU resources and storage resources, creating a 3D_MV instance, setting an MV resolution, setting an MV type, etc. In this way, through the frame interpolation preprocessing, the frame interpolation processing can be more efficient, so that the display frame rate of the first application can be improved.

[0118] In an optional scenario, the MV resolution and the MV type can be default. In addition, the MV resolution and the MV type can also be adjusted according to the indication information of the application developer. The MV type can include an interpolation mode and an extrapolation mode.

[0119] In another possible scenario, the MV resolution and the MV type can be determined according to the indication information of the application developer. For example, the operating system of the electronic device can obtain the MV resolution and the MV type from a configuration file of an application program development package.

[0120] In yet another possible scenario, the MV resolution and the MV type can be calculated according to the indication information of the application developer. For example, the operating system of the electronic device can obtain the configuration rule of the MV resolution and the MV type from the configuration file of the application program development package, and then determine or adjust the MV resolution and the MV type based on the configuration rule.

[0121] The partial process of the interpolation processing is introduced below through steps 502 to 505.

[0122] Step 502: The electronic device renders the N-1th frame image, and displays the rendered N-1th frame image.

[0123] Wherein, N is a positive integer greater than or equal to 2.

[0124] Specifically, the electronic device sends the rendered N-1th frame image to the display unit 160 as shown in FIG. 1 for display.

[0125] It can be understood that the rendering of the N-1th frame image can be implemented by the processor 130 as shown in FIG. 1, which can be a GPU or the like.

[0126] In addition, the electronic device renders the N-1th frame image in a native rendering manner. It can also be understood that the N-1th frame image is a native rendering frame.

[0127] Step 503: During the rendering process of the N-1th frame image, the electronic device stores the 3D data of the preset object in the N-1th frame image according to the pre-labeling of the preset object.

[0128] Wherein, the pre-labeled preset object can be achieved by the process as shown in FIG. 4. During the rendering process, by calling the function of the first attribute integrated in the application plug-in, the 3D data corresponding to the pre-labeled preset object can be obtained.

[0129] For example, the preset object is a game character, and the electronic device can store the 3D data of the game character in the N-1th frame image, which can be at least one of 3D geometric vertex data, 3D polygon mesh data, 3D geometric edge data, etc.

[0130] For example, the electronic device can store the 3D data of the preset object through a preset buffer. It can be understood that during the execution of step 502, step 503 can be processed in parallel through another process or thread, which can be achieved in parallel to store the 3D data of the preset object in the N-1th frame image during the native rendering process of the N-1th frame image, and will not affect the native rendering process of the N-1th frame image.

[0131] Based on the MV type can include interpolation mode and extrapolation mode, which are introduced as follows:

[0132] (I) Interpolation mode

[0133] Step 504A: The electronic device renders the Nth frame image.

[0134] It can be understood that the rendering of the Nth frame image can also be implemented by the processor 130 as shown in FIG. 1, which can be a GPU or the like.

[0135] In addition, the electronic device renders the Nth frame image in a native rendering mode. It can also be understood that the Nth frame image is a native rendering frame, and the Nth frame image is the adjacent next frame of the N-1th frame image.

[0136] Step 505A: During the rendering process of the Nth frame image, the electronic device stores the 3D data of the preset object in the Nth frame image according to the pre-labeling of the preset object.

[0137] The pre-labeled preset object can be obtained by the process as shown in FIG. 4. During the rendering process, the function of the first attribute integrated in the application plug-in can be called to obtain the 3D data corresponding to the pre-labeled preset object.

[0138] For example, the preset object is a game character, and the electronic device can store the 3D data of the game character in the Nth frame image, which can be at least one of 3D geometric vertex data, 3D polygon mesh data, and 3D geometric edge data.

[0139] For example, the electronic device can store the 3D data of the preset object through a preset buffer. The preset buffer for storing the 3D data of the preset object in the Nth frame image can be the same as the preset buffer for storing the 3D data of the preset object in the N-1th frame image.

[0140] It can be understood that during the execution of step 504A, step 505A can be processed in parallel through another process or thread, which can be implemented to store the 3D data of the preset object in the Nth frame image in parallel during the native rendering process of the Nth frame image, and will not affect the native rendering process of the Nth frame image.

[0141] Step 506A: The electronic device obtains and saves the frame information of the N-1th frame image and the Nth frame image. The frame information can include but is not limited to 2D texture map, depth image, camera parameter, etc.

[0142] It should be noted that the electronic device can also obtain the frame information of the N-1th frame and the frame information of the Nth frame respectively, and the specific acquisition order of the frame information of the N-1th frame and the Nth frame is not limited in the present application.

[0143] Step 507A: The electronic device calculates 3D_MV based on 3D data of the preset object in the N-1th frame image and 3D data of the preset object in the Nth frame image.

[0144] In combination with FIG. 2, when the game application in the application layer issues a frame prediction command, the frame insertion service in the application framework layer can calculate 3D_MV through a 3D_MV instance created by calling an interface provided by the 3D_MV module. The frame prediction command can be generated in response to the high frame rate function being set to an enabled state.

[0145] For example, FIG. 6 is a flowchart of calculating 3D_MV provided by an embodiment of the present application. In the flowchart, a multi-level mapping hash algorithm can be used to determine 3D_MV. Specifically, the multi-level hash algorithm can include registration of three levels of drawcall commands, draw instances, and geometric vertices, so as to obtain a high-precision motion vector. As can be seen from FIG. 6, the process of calculating 3D_MV can include the following levels:

[0146] (1) First level: Based on the drawcall command, hash generation is performed to obtain a drawcall hash list of the N-1th frame image and a drawcall hash list of the Nth frame image, respectively. The drawcall command can include a plurality of draw commands. For example, the 3D data of the image can be divided into different draw parts according to types, and each part corresponds to a different draw command. The types can include characters, landscapes, and the like.

[0147] Based on the drawcall hash list of the N-1th frame image and the drawcall hash list of the Nth frame image, hash matching is performed. By judging whether the hash is uniquely matched, the uniquely matched hash in the two drawcall hash lists is determined, so that 3D_MV can be calculated based on the 3D data corresponding to the uniquely matched hash by using a sampling method. In addition, for the remaining 3D data other than the uniquely matched drawcall hash, the second level processing can be performed.

[0148] (2) The second level: based on the instance drawing, the hash generation is performed to obtain the instance hash list of the N-1 frame image and the instance hash list of the N frame image respectively; wherein, the instance drawing can include multiple instances. For example, the person can be divided into head, body, limbs, skin and other instances. It can be understood that the instance hash list of the N-1 frame image and the instance hash list of the N frame image are obtained based on the remaining 3D data after the first level processing.

[0149] Based on the instance hash list of the N-1 frame image and the instance hash list of the N frame image, the hash matching is performed, and by judging whether the hash is uniquely matched, the uniquely matched hash in the two instance hash lists is determined, so that the 3D_MV can be calculated based on the 3D data corresponding to the uniquely matched hash in a sampling manner. In addition, for the remaining 3D data (i.e. the unmatched instance list) other than the uniquely matched instance hash, the third level processing can be performed.

[0150] (3) The third level: taking the 3D data as the vertex data as an example, based on the remaining instance which is not uniquely matched in the first level and the second level, the sampling of the remaining instance can be performed to obtain the sampling vertex, and then the vertex list of the N-1 frame image and the vertex list of the N frame image are obtained.

[0151] Based on the spatial distance of the vertex included in the vertex list of the N-1 frame image and the vertex list of the N frame image, the calculation and matching are performed. For example, the vertex 1 in the N-1 frame image can be matched to the vertex 2 and the vertex 3 in the N frame image, and the spatial distance between the vertex 1 and the vertex 2 is calculated to be less than the spatial distance between the vertex 1 and the vertex 3, then the vertex 1 and the vertex 2 can be uniquely matched.

[0152] Then, based on the spatial distance, the vertex unique matching result is obtained, and based on this part of the 3D data, the 3D_MV is calculated in a sampling manner.

[0153] Wherein, the vertex can be understood as the smallest granularity matching in the image frame.

[0154] In this way, through the multi-level hash matching algorithm, the calculation efficiency of the 3D_MV can be improved.

[0155] Step 508A: The electronic device calculates 2D_MV based on the 2D texture map of the N-1th frame image and the 2D texture map of the Nth frame image.

[0156] It should be noted that the execution order of step 508A is not limited in the embodiments of the present application, and the processing can be performed at any time after step 506A and before step 509A, and can also be processed in parallel with step 507A, etc.

[0157] Step 509A: The electronic device generates the Mth frame image based on 3D_MV and 2D_MV. The Mth frame image is also understood as a predicted frame, etc.

[0158] In an optional embodiment, the electronic device can determine a weight factor of 3D_MV and a weight factor of 2D_MV, weight fuse 3D_MV and 2D_MV according to the weight factor of 3D_MV and the weight factor of 2D_MV to obtain fused MV, and then perform motion compensation according to the fused MV to generate the Mth frame image.

[0159] For example, forward mapping and backward mapping can be performed through 3D_MV to obtain a 3D error map, linear conversion is performed on the 3D error map to obtain a weight weight of 3D_MV, and forward mapping and backward mapping are performed through 2D_MV to obtain a 2D error map, and then linear conversion is performed on the 2D error map to obtain a weight weight of 2D_MV.

[0160] For another example, the weight weights of 3D_MV and 2D_MV can be manually set. Optionally, the weight weight corresponding to 3D_MV can be set to be greater than the weight weight corresponding to 2D_MV.

[0161] Step 510A: The electronic device displays the rendered Mth frame image.

[0162] The Mth frame image is only used for differentiation and is used to represent a preset frame generated based on the N-1th frame image and the Nth frame image.

[0163] Specifically, the electronic device sends the rendered Mth frame image to the display unit 160 shown in FIG. 1 for display.

[0164] Step 511A: The electronic device displays the rendered Nth frame image.

[0165] N is a positive integer greater than or equal to 2.

[0166] Specifically, the electronic device sends the rendered Nth frame image to the display unit 160 shown in FIG. 1 for display.

[0167] It can be understood that in the interpolation mode, the generated Mth frame image is displayed between the (N-1)th frame image and the Nth frame image. Referring to FIG. 7, a timing diagram of image processing in the interpolation mode provided by an embodiment of the present application is shown. For example, the (N-1)th frame image is displayed at time t1A, the Mth frame image can be displayed at time t2A, and the Nth frame image can be displayed at time t3A; wherein t1A is earlier than t2A, and t2A is earlier than t3A. It can be understood that the (N-1)th frame image is obtained by rendering the (N-1)th frame image before it is displayed at time t1A, the Mth frame image is generated based on the rendered (N-1)th frame image and the Nth frame image before it is displayed at time t2A, and the Nth frame image is obtained by rendering the Nth frame image before it is displayed at time t3A.

[0168] (II) Extrapolation mode

[0169] Step 504B: The electronic device renders the Nth frame image and displays the rendered Nth frame image.

[0170] Wherein, N is a positive integer greater than or equal to 2.

[0171] Specifically, the electronic device displays the rendered Nth frame image to the display unit 160 as shown in FIG. 1.

[0172] It can be understood that the rendering of the Nth frame image can also be implemented by the processor 130 as shown in FIG. 1, which can be a GPU or the like.

[0173] In addition, the electronic device renders the Nth frame image in a native rendering mode. It can also be understood that the (N-1)th frame image is a native rendering frame.

[0174] Step 505B: During the rendering process of the Nth frame image, the electronic device stores the 3D data of the preset object in the Nth frame image according to the pre-labeling of the preset object.

[0175] Wherein, the pre-labeled preset object can be obtained by the process as shown in FIG. 4. During the rendering process, by calling the function of the first attribute integrated in the application plug-in, the 3D data corresponding to the pre-labeled preset object can be obtained.

[0176] For example, the preset object is a game character, and the electronic device can store the 3D data of the game character in the Nth frame image, which can be at least one of 3D geometric vertex data, 3D polygon mesh data, and 3D geometric edge data.

[0177] Exemplarily, the electronic device can store the 3D data of the preset object through a preset buffer. The preset buffer storing the 3D data of the preset object of the Nth frame image can be the same as the preset buffer storing the 3D data of the preset object of the (N-1)th frame image.

[0178] It can be understood that in the process of performing step 504B, step 505B can be processed in parallel through another process or thread, can be implemented in parallel to store the 3D data of the preset object in the Nth frame image in the process of performing native rendering of the Nth frame image, and will not affect the native rendering process of the Nth frame image.

[0179] Step 506B: The electronic device acquires and saves the frame information of the (N-1)th frame image and the Nth frame image. The frame information includes but is not limited to 2D texture map, depth image, camera parameter, etc.

[0180] It should be noted that the electronic device can also acquire the frame information of the (N-1)th frame and the frame information of the Nth frame respectively, and the specific acquisition order of the frame information of the (N-1)th frame and the Nth frame is not limited in the present application.

[0181] Step 507B: The electronic device calculates 3D_MV based on the 3D data of the preset object in the (N-1)th frame image and the 3D data of the preset object in the Nth frame image.

[0182] In combination with FIG. 2, when the game application of the application layer issues a frame prediction command, the frame insertion service of the application framework layer can calculate 3D_MV through the created 3D_MV instance by calling the interface provided by the 3D_MV module. The frame prediction command can be generated by the electronic device in response to setting the high frame rate function to an enabled state.

[0183] Exemplarily, FIG. 6 is a flowchart of calculating 3D_MV provided by an embodiment of the present application. In the flowchart, the registration of the drawcall command, the instance, and the vertex is realized through a multi-level mapping hash algorithm, so as to obtain a high-precision motion vector. As can be seen from FIG. 6, in the process of calculating 3D_MV, the following levels can be included:

[0184] (1) The first level: based on the drawcall command, hash generation is performed to obtain the drawcall hash list of the (N-1)th frame image and the drawcall hash list of the Nth frame image respectively; wherein the drawcall command can include a plurality of draw commands. For example, the 3D data of the image can be divided into different draw parts according to types, and each part corresponds to a different draw command; the types can include characters, landscapes, etc.

[0185] Hash matching is performed based on the drawcall hash list of the N-1th frame image and the drawcall hash list of the Nth frame image, and by judging whether the hash is uniquely matched, the uniquely matched hash in the two drawcall hash lists is determined, so that 3D_MV can be calculated in a sampling manner based on the 3D data corresponding to the uniquely matched hash. In addition, for the remaining 3D data other than the uniquely matched drawcall hash, a second level of processing can be performed.

[0186] (2) Second level: based on the drawing instance, hash generation is performed to obtain the instance hash list of the N-1th frame image and the instance hash list of the Nth frame image respectively; wherein the drawing instance can include multiple instances. For example, a person can be divided into head, body, limbs, skin and other instances. It can be understood that the instance hash list of the N-1th frame image and the instance hash list of the Nth frame image are obtained based on the remaining 3D data after the first level of processing.

[0187] Hash matching is performed based on the instance hash list of the N-1th frame image and the instance hash list of the Nth frame image, and by judging whether the hash is uniquely matched, the uniquely matched hash in the two instance hash lists is determined, so that 3D_MV can be calculated in a sampling manner based on the 3D data corresponding to the uniquely matched hash. In addition, for the remaining 3D data (i.e. the unmatched instance list) other than the uniquely matched instance hash, a third level of processing can be performed.

[0188] (3) Third level: taking vertex data as an example, based on the remaining instances that are not uniquely matched in the first level and the second level, the remaining instances can be sampled to obtain sampled vertices, and then the vertex list of the N-1th frame image and the vertex list of the Nth frame image are obtained.

[0189] Based on the spatial distance of the vertex included in the vertex list of the N-1th frame image and the vertex list of the Nth frame image, calculation and matching are performed. For example, vertex 1 in the N-1th frame image can be matched to vertex 2 and vertex 3 in the Nth frame image, and the spatial distance between vertex 1 and vertex 2 is calculated to be less than the spatial distance between vertex 1 and vertex 3, then vertex 1 and vertex 2 can be uniquely matched.

[0190] Then, based on the spatial distance, the vertex unique matching result is obtained, and then the 3D_MV is calculated by sampling based on the part of 3D data.

[0191] The vertex can be understood as the minimum granularity matching in the image frame.

[0192] In this way, the calculation efficiency of the 3D_MV can be improved by the multi-level hash matching algorithm.

[0193] Step 508B: The electronic device calculates the 2D_MV based on the 2D texture map of the N-1th image and the 2D texture map of the Nth image.

[0194] It should be noted that the execution order of step 508B is not limited in the embodiments of the present application, and the processing can be performed at any time after step 506B and before step 509B, and the processing can also be performed in parallel with step 507B, etc.

[0195] Step 509B: The electronic device generates the Pth image based on the 3D_MV and the 2D_MV. The Pth image is also understood as a predicted frame, etc.

[0196] In an optional embodiment, the electronic device can perform weighted fusion on the 3D_MV and the 2D_MV to obtain the fused MV, and then perform motion compensation according to the fused MV to generate the Pth image.

[0197] For example, the 3D_MV can be used for forward mapping and backward mapping to obtain a 3D error map, the 3D error map is linearly converted to obtain the weighted weight of the 3D_MV, and the 2D_MV is used for forward mapping and backward mapping to obtain a 2D error map, and then the 2D error map is linearly converted to obtain the weighted weight of the 2D_MV.

[0198] Another example is that the weighted weights of the 3D_MV and the 2D_MV can be manually set. Optionally, the weighted weight corresponding to the 3D_MV can be set to be greater than the weighted weight corresponding to the 2D_MV.

[0199] Step 510B: The electronic device displays the rendered Pth image.

[0200] The Pth image is only used for differentiation, and is used to represent a preset frame generated based on the N-1th image and the Nth image.

[0201] Specifically, the electronic device sends the rendered Pth image to the display unit 160 as shown in FIG. 1 for display.

[0202] It can be understood that in the extrapolation mode, the generated P-th frame image is sent for display after the (N-1)-th frame image and the N-th frame image are displayed. Referring to FIG. 8, a timing diagram of image processing in the extrapolation mode provided by an embodiment of the present application is shown. For example, the (N-1)-th frame image is sent for display at t1B, the N-th frame image can be sent for display at t2B, and the P-th frame image can be sent for display at t3B; wherein t1B is earlier than t2B, and t2B is earlier than t3B. It can be understood that the (N-1)-th frame image is obtained by rendering the (N-1)-th frame image before the (N-1)-th frame image is sent for display at t1B, the N-th frame image is obtained by rendering the N-th frame image before the N-th frame image is sent for display at t2B, and the P-th frame image is generated based on the rendered (N-1)-th frame image and the N-th frame image before the P-th frame image is sent for display at t3B.

[0203] By the method provided by the embodiment of the present application, in the process of performing the frame interpolation processing, the 3D data of the preset object pre-marked in the two adjacent native rendering frames is combined for calculating the MV, and the 3D_MV and the 2D_MV can be fused to obtain the MV for generating the prediction frame, so that the quality of the generated prediction frame can be improved, and the probability of the problems such as ghosting and virtual shadow in the prediction frame can be reduced.

[0204] FIG. 9 is another possible flow diagram of an image processing method provided by an embodiment of the present application. Referring to FIG. 9, the method can be applied to an electronic device, and can include the following steps:

[0205] In step 901, a preset event is detected, wherein the preset event is used to indicate that the frame interpolation function is in an enabled state.

[0206] Optionally, the detection of the preset event can be implemented as: detecting a first operation on a first control, wherein the first control is used to manage the frame interpolation function, and the first operation is used to enable the frame interpolation function. In combination with FIG. 3, the first control can be a high frame rate control as shown in interface 301, and the first operation can be setting the high frame rate control to the enabled state as shown in interface 302.

[0207] In step 902, in response to the preset event, first 3D data of a preset object in a first native rendering frame is obtained, a first 2D texture map of the first native rendering frame is obtained, second 3D data of the preset object in a second native rendering frame is obtained, and a second 2D texture map of the second native rendering frame is obtained, wherein the second native rendering frame is a next frame adjacent to the first native rendering frame.

[0208] For example, the first 3D data or the second 3D data includes at least one of the following data: 3D geometric vertex data, 3D polygon mesh data, and 3D geometric edge data.

[0209] Step 903, determining a first motion vector based on the first 3D data and the second 3D data, and determining a second motion vector based on the first 2D texture map and the second 2D texture map. The first motion vector can be understood as the 3D_MV in the foregoing embodiments, the second motion vector can be understood as the 2D_MV in the foregoing embodiments, and the third motion vector can be understood as the MV after fusion in the foregoing embodiments.

[0210] Optionally, the determining the first motion vector based on the first 3D data and the second 3D data can be implemented as: determining the first motion vector based on the first 3D data and the second 3D data by using a multi-level hash algorithm. For example, refer to FIG. 6 and the corresponding introduction.

[0211] Step 904, generating a predicted frame according to the first motion vector and the second motion vector. The predicted frame can be understood as the Mth image in the scene of using the interpolation mode for the frame interpolation processing, or the Pth image in the scene of using the extrapolation mode for the frame interpolation processing in the foregoing embodiments.

[0212] For example, the generating the predicted frame according to the first motion vector and the second motion vector can be implemented as: determining a first weight factor of the first motion vector and a second weight factor of the second motion vector; performing weighted fusion on the first motion vector and the second motion vector according to the first weight factor and the second weight factor to obtain a third motion vector; and generating the predicted frame according to the third motion vector.

[0213] In a possible design, the display order between the predicted frame and the first native rendering frame and the second native rendering frame is: displaying the first native rendering frame; displaying the predicted frame; and displaying the second native rendering frame. It can be understood that this design is to perform the frame interpolation processing by using the interpolation mode.

[0214] In another possible design, the display order between the predicted frame and the first native rendering frame and the second native rendering frame is: displaying the first native rendering frame; displaying the second native rendering frame; and displaying the predicted frame. It can be understood that this design is to perform the frame interpolation processing by using the extrapolation mode.

[0215] The specific implementation process of steps 901 to 904 in FIG. 9 can refer to the introduction in the foregoing embodiments, which will not be repeated here.

[0216] FIG. 10 is a possible structural example of an image processing apparatus provided by the embodiments of the present application. Referring to FIG. 10, the image processing apparatus can include a processing unit 1001.

[0217] In an embodiment, the image processing apparatus can be used to implement the method performed by any of the electronic devices in the above embodiments. The image processing apparatus can be the electronic device itself, or a chip or chip set or a part of a chip in the electronic device used to perform the functions of the related method. The processing unit 1001 is configured to perform the following processing: detecting a preset event, wherein the preset event is used to indicate that the frame interpolation function is in an open state; in response to the preset event, obtaining first 3D data of a preset object in a first native rendering frame, obtaining a first 2D texture map of the first native rendering frame, obtaining second 3D data of the preset object in a second native rendering frame, and obtaining a second 2D texture map of the second native rendering frame, wherein the second native rendering frame is a neighboring next frame of the first native rendering frame; determining a first motion vector based on the first 3D data and the second 3D data; determining a second motion vector based on the first 2D texture map and the second 2D texture map; and generating a predicted frame according to the first motion vector and the second motion vector.

[0218] In a possible design, the image processing apparatus can further include a display unit. The display unit can be configured to display the predicted frame in an interpolation mode or display the predicted frame in an extrapolation mode. For details, refer to the description of the foregoing embodiments, which will not be repeated here.

[0219] The division of units (function modules) in the embodiments of the present application is illustrative, and is merely a logical function division. In actual implementation, another division manner can be used. In addition, the units in each embodiment of the present application can be integrated in one processor, or can be separate physical units, or two or more units can be integrated in one module. The integrated units can be implemented in the form of hardware or in the form of software function modules. It can be understood that the functions or implementation of the units in the embodiments of the present application can be further referred to the related description of the method embodiments.

[0220] FIG. 11 is a possible structural example of an apparatus provided in an embodiment of the present application, which can be used to implement the functions of the electronic device in the above embodiments, and thus can also achieve the beneficial effects possessed by the above embodiments.

[0221] Referring to FIG. 11, the apparatus 1100 can include a processor 1101, a memory 1102, and a communication interface 1103. The processor 1101, the memory 1102, and the communication interface 1103 are coupled to each other. Optionally, the memory 1102 can be used to store instructions executed by the processor 1101 or to store input data required for the processor 1101 to run instructions or to store data generated after the processor 1101 runs instructions. The communication interface 1103 can be a transceiver or an input / output interface.

[0222] Optionally, referring to FIG. 11, the processor 1101, the memory 1102 and the communication interface 1103 are connected with each other through a bus 1104. The bus 1104 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, only one thick line is used in FIG. 11, but it does not mean that there is only one bus or only one type of bus.

[0223] When the device 1100 is applied to an electronic device, the processor 1101 can be used to implement the processing function of the electronic device in the above embodiments. For example, the processor 1101 can be used to execute the actions of the processing unit 1001 when the image processing apparatus is applied to the electronic device. The communication interface 1103 can be used to implement the transceiving function of the electronic device in the above embodiments, which will not be described in detail.

[0224] It can be understood that the processor 1101 in the embodiments of the present application can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor can be a microprocessor or any conventional processor.

[0225] The memory 1102 in FIG. 11 of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically EPROM (EEPROM) or a flash memory. It should be noted that the memory of the system and method described herein is intended to include but not limited to these and any other suitable types of memory.

[0226] Based on the above embodiments, the present application further provides a computer program product comprising instructions, which, when executed on a computer, cause the computer to carry out the methods described in the embodiments of the present application.

[0227] Based on the above embodiments, the present application further provides a computer readable storage medium, wherein a computer program is stored in the computer readable storage medium, and when the computer program is executed on a computer, the computer is caused to carry out the methods described in the embodiments of the present application.

[0228] Based on the above embodiments, the present application further provides a chip for reading a computer program stored in a memory, and implementing the methods described in the embodiments of the present application.

[0229] Based on the above embodiments, the present application provides a chip system, which comprises a processor for supporting a computer device to implement the methods described in the embodiments of the present application. In a possible design, the chip system further comprises a memory for storing the necessary programs and data of the computer device. The chip system can be composed of a chip, or can include a chip and other discrete devices.

[0230] Those skilled in the art should understand that the embodiments of the present application can be provided as methods, systems or computer program products. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage etc.) containing computer-usable program code.

[0231] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing devices to produce a machine, so that the instructions executed by the computer or other programmable data processing devices produce the functions specified in one or more flows in the flowcharts and / or one or more blocks in the block diagrams.

[0232] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart or flowsheets and / or block or blocks of the block diagrams.

[0233] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart or flowsheets and / or block or blocks of the block diagrams.

[0234] Obviously, numerous modifications and variations of the present application are possible in light of the above teachings. It is therefore to be understood that within the scope of the claims and their equivalents, the application can be practiced otherwise than as specifically described.

Claims

1. An image processing method, characterized by, The method is applied to an electronic device, and the method comprises: detecting a preset event, wherein the preset event is used to indicate that an inter-frame insertion function is in an open state; in response to the preset event, acquiring first 3D data of a preset object in a first native rendering frame, acquiring a first 2D texture map of the first native rendering frame, acquiring second 3D data of the preset object in a second native rendering frame, and acquiring a second 2D texture map of the second native rendering frame, wherein the second native rendering frame is a next frame adjacent to the first native rendering frame; determining a first motion vector based on the first 3D data and the second 3D data, and determining a second motion vector based on the first 2D texture map and the second 2D texture map; generating a predicted frame according to the first motion vector and the second motion vector.

2. The method of claim 1, wherein, The method further comprises: displaying the first native rendering frame; displaying the predicted frame; displaying the second native rendering frame.

3. The method of claim 1, wherein, The method further comprises: displaying the first native rendering frame; displaying the second native rendering frame; displaying the predicted frame.

4. The method according to any one of claims 1 to 3, characterized in that, The method further comprises: displaying the first native rendering frame; 5. The method according to any one of claims 1 to 4, characterized in that, displaying the second native rendering frame; displaying the predicted frame. The determining of the first motion vector based on the first 3D data and the second 3D data comprises: determining the first motion vector based on the first 3D data and the second 3D data by using a multi-level hash algorithm.

6. The method according to any one of claims 1 to 5, characterized in that, The generating of the predicted frame according to the first motion vector and the second motion vector comprises: determining a first weight factor of the first motion vector and a second weight factor of the second motion vector; 7. The method according to any one of claims 1 to 6, characterized in that, performing weighted fusion on the first motion vector and the second motion vector according to the first weight factor and the second weight factor to obtain a third motion vector; 8. An electronic device, comprising: generating the predicted frame according to the third motion vector.

9. A chip system, characterized by The detecting of the preset event comprises:

10. A computer-readable storage medium, characterized in that, detecting a first operation on a first control, wherein the first control is used to manage the inter-frame insertion function, and the first operation is used to open the inter-frame insertion function.

11. A computer program product, characterised in that, The first 3D data or the second 3D data comprises at least one of the following data: 3D geometric vertex data, 3D polygon mesh data, and 3D geometric edge data. The chip system comprises a processing circuit, a storage medium, and instructions stored in the storage medium; when the instructions are executed by the processing circuit, the method according to any one of claims 1-7 is implemented. The chip system comprises a processing circuit, a storage medium, and instructions stored in the storage medium; when the instructions are executed by the processing circuit, the method according to any one of claims 1-7 is implemented. The computer readable storage medium stores instructions, which, when executed on a computer, cause the computer to perform the method according to any one of claims 1-7. The computer program product, when executed on a computer, causes the computer to perform the method according to any one of claims 1-7.

Images