Image processing method and electronic device

Abstract

The application discloses an image processing method and an electronic device, and belongs to the technical field of image processing. The specific scheme comprises the following steps: acquiring first display data, wherein the first display data is the synthesis data of multiple layers; performing first processing on the first display data to obtain second display data, performing second processing on the second display data to obtain third display data; and displaying an image picture according to the third display data; wherein the first processing comprises frame loss processing on odd frames or even frames, and the second processing comprises inserting a third image frame between a first image frame and a second image frame, wherein the third image frame is an image frame determined according to the first image frame and the second image frame, and the first image frame and the second image frame are adjacent image frames in the second display data.

Description

Technical Field

[0001] This application belongs to the field of image processing technology, specifically relating to an image processing method and an electronic device. Background Technology

[0002] With the rise of live streaming, more and more users enjoy using the bullet screen feature to interact with the streamer. To ensure smooth bullet screen scrolling, the frame rate of the bullet screen layer is usually higher than that of the video layer.

[0003] In related technologies, after receiving the bullet screen layer and video layer from the application, the system chip of the electronic device can first unify the frame rate of the video layer and the bullet screen layer by repeatedly reading the image frames of the video layer. Then, the two layers with the unified frame rate are merged and output to an independent display chip for frame interpolation processing to obtain image display data. Finally, the image display data is transmitted to the display screen for display.

[0004] However, this frame interpolation method increases the processing burden on the discrete display chip and results in higher system power consumption. Summary of the Invention

[0005] The purpose of this application is to provide an image processing method and electronic device that can solve the problem of poor image smoothness when using system chips for image frame interpolation.

[0006] In a first aspect, embodiments of this application provide an image processing method, the method comprising: acquiring first display data, the first display data being composite data of multiple layers; performing a first processing on the first display data to obtain second display data; performing a second processing on the second display data to obtain third display data; and displaying an image according to the third display data; wherein the first processing includes frame dropping processing for odd-numbered or even-numbered frames, and the second processing includes inserting a third image frame between a first image frame and a second image frame, the third image frame being an image frame determined based on the first image frame and the second image frame, and the first image frame and the second image frame being adjacent image frames in the second display data.

[0007] Secondly, embodiments of this application provide an image processing apparatus, including: an acquisition module, a processing module, and a display module; the acquisition module is used to acquire first display data, the first display data being composite data of multiple layers; the processing module is used to perform a first processing on the first display data to obtain second display data, and perform a second processing on the second display data to obtain third display data; the display module is used to display an image frame according to the third display data; wherein, the first processing includes frame dropping processing for odd-numbered or even-numbered frames, the second processing includes inserting a third image frame between a first image frame and a second image frame, the third image frame being an image frame determined based on the first image frame and the second image frame, and the first image frame and the second image frame being adjacent image frames in the second display data.

[0008] Thirdly, embodiments of this application provide an electronic device, which includes a system chip, a discrete display chip, a display, and a data transmission interface; the system chip is used to acquire first display data, perform a first processing on the first display data to obtain second display data, wherein the first display data is composite data of multiple layers; the data transmission interface is used to transmit the second display data from the system chip to the discrete display chip; the discrete display chip is used to perform a second processing on the second display data to obtain third display data; the display is used to display an image screen according to the third display data; wherein the first processing includes frame dropping processing for odd or even frames, and the second processing includes inserting a third image frame between the first image frame and the second image frame, wherein the third image frame is an image frame determined according to the first image frame and the second image frame, and the first image frame and the second image frame are adjacent image frames in the second display data.

[0009] Fourthly, embodiments of this application provide an electronic device including a processor and a memory, the memory storing programs or instructions executable on the processor, the programs or instructions, when executed by the processor, implementing the steps of the method described in the first aspect.

[0010] Fifthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0011] In a sixth aspect, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the method as described in the first aspect.

[0012] In a seventh aspect, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the method described in the first aspect.

[0013] In this embodiment, first display data, which is composite data of multiple layers, can be obtained. The first display data undergoes a first processing step to obtain second display data, and the second display data undergoes a second processing step to obtain third display data. An image is displayed according to the third display data. The first processing step includes dropping frames for odd or even-numbered frames, and the second processing step includes inserting a third image frame between the first and second image frames. The third image frame is an image frame determined based on the first and second image frames, and the first and second image frames are adjacent image frames in the second display data. This approach achieves several advantages. Firstly, inserting a third image frame between the first and second image frames, based on the first and second image frames, ensures a smooth transition between adjacent image frames, thus improving the smoothness of the image. Secondly, by first dropping frames for odd or even-numbered frames of the first display data and then inserting frames, the processing burden of the second processing step is reduced, thereby lowering system power consumption and extending device standby time. Attached Figure Description

[0014] Figure 1 This is one of the structural schematic diagrams of the electronic device provided in the embodiments of this application;

[0015] Figure 2 This is a schematic diagram of the unified video layer data and bullet screen layer data provided in the embodiments of this application;

[0016] Figure 3 This is one of the schematic flowcharts of the image processing method provided in the embodiments of this application;

[0017] Figure 4 This is a schematic diagram of the data transmission process of the image processing method provided in the embodiments of this application;

[0018] Figure 5 This is a second schematic flowchart of the image processing method provided in the embodiments of this application;

[0019] Figure 6 This is one of the data processing flow diagrams provided in the embodiments of this application;

[0020] Figure 7 This is the second schematic diagram of the data processing flow provided in the embodiments of this application;

[0021] Figure 8 This is a schematic diagram of the structure of the image processing apparatus provided in the embodiments of this application;

[0022] Figure 9 This is a second schematic diagram of the structure of the electronic device provided in the embodiments of this application;

[0023] Figure 10 This is a hardware schematic diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0024] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0025] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0026] The following is a detailed explanation of the technical terms used in the embodiments of this application.

[0027] System on Chip (SoC): A system is a combination of multiple integrated circuits with specific functions. For example, integrated circuits may include modules such as central processing units, image processors, signal processors, memory, and modems.

[0028] System service (SurfaceFlinger): This refers to a system service in the Android operating system that can receive Surface information drawn by multiple applications, generate image data to be displayed on the screen, and pass the image data to the hardware display.

[0029] Hardware Composer (HWC): Provides hardware support for the SurfaceFlinger service. After SurfaceFlinger provides a list of layers to the HWC, the HWC can process the layer data according to its hardware capabilities.

[0030] The Direct Rendering Manager (DRM) is a subsystem of the Linux kernel that user-space programs can use to send commands and data to the graphics processor and perform operations such as configuring display mode settings.

[0031] Mobile Industry Processor Interface (MIPI): Used to standardize hardware and software interfaces, and in this embodiment, it is used to represent the transmitted display data.

[0032] Display Serial Interface (DSI): A protocol interface used for displaying data under the MIPI protocol.

[0033] Frame Pacing library: Enables smooth rendering and frame synchronization of image data through code definition.

[0034] General purpose input / output (GPIO): Used to transmit control commands.

[0035] Receiver (RX): Used to receive the content to be displayed; the number of ports can be increased as needed.

[0036] Transmitter (Transport, TX): Used to transmit content that has been processed by the independent display chip. The number of ports can be increased as needed.

[0037] Discrete graphics IP: The internal functional modules of a discrete graphics chip. The number is not fixed and can be selected according to needs.

[0038] like Figure 1 As shown in the figure, this application provides an electronic device, which includes a system chip 100, a display 200, a standalone display chip 300, and a data transmission interface 400.

[0039] The system chip 100 may include an application 101, a first layer path 102, a second layer path 103, a system service 104, a hardware compositor 105, and a direct rendering manager 106.

[0040] The data transmission interface 400 may include a data transmission port DSP0 and a control port GPIO1 of the system chip 100, and a data reception port DSPRX and a control port GPIO2 of the independent display chip 300. The data transmission port DSP0 can be used to send layer data, and the data reception port DSPRX can be used to receive layer data. The control ports GPIO1 and GPIO2 can be used to transmit control commands from the system chip 100 to the independent display chip 300.

[0041] The independent display chip 300 may include an independent display IP module, display path 1, display path 2, and data transmission port DSP TX.

[0042] The system chip 100 is used to acquire first display data, perform first processing on the first display data to obtain second display data, where the first display data is composite data of multiple layers. A data transmission interface is used to transmit the second display data from the system chip to the independent display chip. The independent display chip 300 is used to perform second processing on the second display data to obtain third display data. The display 200 is used to display an image according to the third display data; wherein the first processing includes frame dropping processing for odd or even frames, and the second processing includes inserting a third image frame between the first and second image frames, where the third image frame is an image frame determined based on the first and second image frames, and the first and second image frames are adjacent image frames in the second display data.

[0043] Specifically, the system chip 100 can obtain video layer data and bullet screen layer data from the application 101. Then, the system chip 100 can send the video layer data to the system service 104 in the Android operating system through the first layer path 102 and send the bullet screen layer data to the system service 104 through the second layer path 103. The system service 104 can perform layer merging processing on the received video layer data and bullet screen layer data to obtain the first display data. Then, the first display data is processed by frame dropping to obtain the second display data. Then, the system service 104 can send the second display data to the hardware compositor 105. After the hardware compositor 105 performs compositing processing, the second display data is transmitted from the direct rendering manager 106 in the form of a MIPI signal through the data transmission port DSP 0 to the data reception port DSP RX. The second display data transmitted to the independent display chip 300 via the data receiving port DSP RX can be directly transmitted to the monitor 200 via display path 1 and data sending port DSP TX. Alternatively, when the system chip 100 sends control commands instructing the independent display to perform independent display processing, the data can be transmitted via display path 2, processed by the independent display IP module through frame interpolation to obtain the third display data, and then transmitted to the monitor 200 via data sending port DSP TX. Finally, the monitor 200 can be used to play video including subtitles.

[0044] It's important to understand that the frame rate of image source data varies depending on the application scenario. For example, if the image source data is traditional movie layer data, its frame rate is 24fps; if it's TV program layer data, its frame rate is 30fps; and if it's bullet screen layer data, its frame rate is 60fps. If the frame rates of the layer data received by system service 104 are different, system service 104 needs to implement frame synchronization processing based on the definition of the frame synchronization code library before performing layer merging processing.

[0045] Specifically, when the system service 104 detects that the frame rate of the image rendered by the application is inconsistent with the refresh rate of the display hardware, the system service 104 can read data from the previous frame buffer. At the hardware level, this means that the display 200 can display the previous image frame.

[0046] For example, such as Figure 2 As shown, if the frame rate of the video layer data is 24fps and the frame rate of the bullet screen layer data is 60fps, then when playing a 24fps video at a screen refresh rate of 60Hz, system service 104 can read the content of buffer "A" in the first 1 / 60s; in the second 1 / 60s, system service 104 still reads the content of buffer "A"; in the third 1 / 60s, system service 104 can read the content of buffer "B"; in the fourth 1 / 60s, system service 104 reads the content of buffer "B"; in the fifth 1 / 60s, system service 104 still reads the content of buffer "B"; in the sixth 1 / 60s, system service 104 can read the content of buffer "C", and so on until all buffers are read. In other words, system service 104 can achieve synchronous reading of video layer data and bullet screen layer data based on a repeating rhythm of two, three, two, three times.

[0047] Continue to refer to Figure 2 If the frame rate of the video layer data is 30fps and the frame rate of the bullet screen layer data is 60fps, then when playing a 30fps video at a screen refresh rate of 60Hz, system service 104 can read the content of buffer "A" in the first 1 / 60s; in the second 1 / 60s, system service 104 still reads the content of buffer "A"; in the third 1 / 60s, system service 104 can read the content of buffer "B"; in the fourth 1 / 60s, system service 104 reads the content of buffer "B"; in the fifth 1 / 60s, system service 104 can read the content of buffer "C"; in the sixth 1 / 60s, system service 104 can still read the content of buffer "C", and so on until all buffers are read. In other words, system service 104 can achieve synchronous reading of video layer data and bullet screen layer data based on a repeating rhythm of two, two, two.

[0048] In this embodiment, since the second display data can be transmitted from the system chip to the independent display chip through the data transmission interface, single-channel transmission of multi-layer data can be realized. Compared with multi-channel transmission, it can not only reduce power consumption, avoid compatibility issues, and reduce chip manufacturing costs, but also eliminate the need for considerations on data transmission synchronization, thus reducing the difficulty of system debugging tasks.

[0049] The image processing method provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0050] The image processing method provided in this application embodiment can be executed by an electronic device or a functional module or entity in an electronic device that can implement the image processing method. The electronic devices mentioned in this application embodiment include, but are not limited to, mobile phones, tablets, computers, cameras, wearable devices, etc. The image processing method provided in this application embodiment is described below using an electronic device as the execution subject as an example.

[0051] like Figure 3 As shown, this application embodiment provides an image processing method, which may include steps 501-503:

[0052] Step 501: Obtain the first display data.

[0053] The first displayed data is a composite data of multiple layers.

[0054] Optionally, the aforementioned multiple layers may include first layer data and second layer data, and the first display data may be composite data of the first layer data and the second layer data.

[0055] Optionally, the first layer data mentioned above can be video layer data, and the second layer data mentioned above can be bullet screen layer data. The source of the video layer data can be any of the following: movies, television, animation, live videos, etc.

[0056] Optionally, the electronic device can use the system service in the aforementioned system chip to merge video layer data and bullet screen layer data with the same frame rate to obtain the first display data.

[0057] Specifically, before the system service performs layer merging, the frame rate of the video layer data is lower than that of the bullet screen layer data. For example, the frame rate of the video layer data can be 24fps or 30fps, while the frame rate of the bullet screen layer data can be 60fps. The electronic device can first increase the frame rate of the video layer data to 60fps by repeatedly reading frames, and then perform layer merging to obtain the first display data with a frame rate of 60fps. The implementation process of repeatedly reading frames can be found in [reference needed]. Figure 2 The relevant descriptions of the corresponding text sections mentioned above will not be repeated here.

[0058] Step 502: Perform a first processing on the first display data to obtain the second display data, and perform a second processing on the second display data to obtain the third display data.

[0059] The first processing includes frame dropping processing for odd or even frames, and the second processing includes inserting a third image frame between the first image frame and the second image frame. The third image frame is an image frame determined based on the first image frame and the second image frame, and the first image frame and the second image frame are adjacent image frames in the second display data.

[0060] Optionally, the electronic device can perform a first processing on the first display data through a system chip to obtain second display data; and perform a second processing on the second display data through a separate display chip to obtain third display data.

[0061] Specifically, the electronic device can perform uniformly spaced frame dropping on the first display data through the system service in the system chip to obtain the second display data. Then, the second display data is transmitted to the independent display chip through the hardware compositor, direct rendering manager and DSIO port in the system chip. The independent display chip can perform frame interpolation on the second display data through the built-in frame interpolation algorithm to obtain the third display data.

[0062] For example, such as Figure 4 As shown, taking a first display data frame rate of 60fps as an example, the discrete display chip 300's discrete graphics IP module includes a motion estimation and motion compensation (MEMC) module 301. System service 104 can obtain second display data with a frame rate of 30fps by processing the dropped frames of the first display data (odd or even frames). Then, system service 104 can sequentially transmit the second display data with a frame rate of 30fps to the discrete display chip 300 via hardware compositor 105, direct rendering manager 106, and DSIO port. After receiving the second display data, the discrete display chip 300 can perform frame interpolation processing on the second display data with a frame rate of 30fps through the MEMC module 301 to obtain a third display data with a frame rate of 60fps.

[0063] Based on the above scheme, since the first display data can be processed by the system chip to obtain the second display data, and the second display data can be processed by the independent display chip to obtain the third display data, the frame rate of the display data can be increased by the frame interpolation processing of the independent display chip while reducing the processing burden of the independent display chip, thereby improving the display effect of the image.

[0064] Step 503: Display the image screen according to the third display data.

[0065] Alternatively, the electronic device may display an image of third-party data on a monitor.

[0066] In this embodiment, on the one hand, a third image frame can be inserted between the first image frame and the second image frame. Since the third image frame is determined based on the first image frame and the second image frame, inserting the third image frame can achieve a smooth transition between adjacent image frames, thereby improving the smoothness of the image. On the other hand, since the odd or even frames of the first display data can be dropped first and then the frame insertion can be performed, the processing burden of the second processing can be reduced, thereby reducing system power consumption and extending the device standby time.

[0067] Optionally, before acquiring the first display data, the electronic device may receive the layer data to be displayed, which includes at least the first layer data; if the layer data to be displayed also includes the second layer data, the electronic device may determine the first display data based on the first layer data and the second layer data; if the layer data to be displayed does not include the second layer data, the electronic device may perform frame interpolation on the first layer data to obtain the fourth display data, and display the image according to the fourth display data.

[0068] Specifically, such as Figure 5 As shown, when an electronic device plays a video through an application, the system service in the electronic device can receive the layer data to be displayed from the application and identify whether the layer data to be displayed includes the second layer data, that is, identify whether the video has bullet comments enabled. If the layer data to be displayed includes the second layer data, the system service can determine the first display data based on the first layer data and the second layer data. If the layer data to be displayed does not include the second layer data, the electronic device can perform frame interpolation on the first layer data to obtain the fourth display data and display the image screen according to the fourth display data.

[0069] For example, consider a first layer of data consisting of 30fps video stream data and a second layer of data consisting of 60fps bullet comment stream data. If the electronic device recognizes that the layer data to be displayed includes both video stream data and bullet comment stream data, it can generate display data containing bullet comments based on the video stream data and the bullet comment stream data. If the electronic device recognizes that the layer data to be displayed only includes video stream data, it can first process the 30fps video stream data into 60fps video stream data through frame interpolation, and then display the image according to the 60fps video stream data.

[0070] Based on the above scheme, since the first display data can be determined based on the first layer data and the second layer data when the layer data to be displayed includes the first layer data and the second layer data, and the first layer data can be processed by frame interpolation to obtain the fourth display data when the layer data to be displayed does not include the second layer data, and the image is displayed according to the fourth display data, it is possible to differentiate between the layer data to be displayed that includes the second layer data and the layer data that does not include the second layer data, thereby improving the image display effect of different video sources in a more targeted manner.

[0071] Optionally, the electronic device determines the first display data based on the first layer data and the second layer data. Specifically, this may include: the electronic device determining the display frame rate of the first layer data and the display frame rate of the second layer data, wherein the display frame rate of the first layer data is the first frame rate and the display frame rate of the second layer data is the second frame rate, and the first frame rate is less than the second frame rate; determining the number of repeated readings based on the first frame rate and the second frame rate, and reading the first layer data according to the number of repeated readings to obtain the third layer data, wherein the display frame rate of the third layer data is the second frame rate; and performing layer merging processing on the second layer data and the third layer data to obtain the first display data.

[0072] Optionally, the first frame rate can be 24fps or 30fps, and the second frame rate can be 60fps.

[0073] For example, such as Figure 6 As shown, taking a first frame rate of 24fps and a second frame rate of 60fps as an example, the electronic device can determine that the display frame rate of the first layer of data is 24fps and the display frame rate of the second layer of data is 60fps. Then, the electronic device can determine the number of times to repeatedly read the first layer of data based on the display frame rates of the first and second layers, and read the first layer of data according to this number of repetitions to obtain the third layer of data with a frame rate of 60fps. Finally, the electronic device can merge the second and third layer of data to obtain the first display data with a frame rate of 60fps. Afterwards, the electronic device can identify whether the first display data has odd or even frames and perform uniform frame dropping to obtain the second display data with a frame rate of 30fps. Then, after receiving the second display data through the independent display chip 300, it performs frame interpolation on the second display data with a frame rate of 30fps to obtain the third display data with a frame rate of 60fps.

[0074] For example, such as Figure 7As shown, taking a first frame rate of 30fps and a second frame rate of 60fps as an example, the electronic device can determine that the display frame rate of the first layer data is 30fps and the display frame rate of the second layer data is 60fps. Then, the electronic device can determine the number of times to repeatedly read the first layer data based on the display frame rates of the first and second layers, and read the first layer data according to this number of repetitions to obtain the third layer data with a frame rate of 60fps. Finally, the electronic device can merge the second and third layer data to obtain the first display data with a frame rate of 60fps. Afterwards, the electronic device can identify whether the first display data has odd or even frames and perform uniform frame dropping to obtain the second display data with a frame rate of 30fps. Then, after receiving the second display data through the independent display chip 300, it performs frame interpolation on the second display data with a frame rate of 30fps to obtain the third display data with a frame rate of 60fps.

[0075] Based on the above scheme, since the number of repeated readings can be determined based on the first frame rate and the second frame rate, and the third layer data can be obtained, this provides a basis for merging the second layer data and the third layer data to obtain the first display data.

[0076] Optionally, the electronic device determines the number of repeated reads based on the first frame rate and the second frame rate, and reads the first layer data according to the number of repeated reads to obtain the third layer data. Specifically, this may include: the electronic device determining the ratio of the second frame rate to the first frame rate; if the ratio is an integer, repeatedly reading each image frame of the first layer data according to the first number to obtain the third layer data, where the first number is the ratio; if the ratio is not an integer, repeatedly reading the third image frame of the first layer data according to the second number, and repeatedly reading the fourth image frame of the first layer data according to the third number to obtain the third layer data, where the second number is N, the third number is N+1, and the ratio is within the range of (N, N+1); wherein the third image frame and the fourth image frame are adjacent image frames in the first layer data, and N is a positive integer.

[0077] For example, continue to refer to Figure 6 Taking a frame rate of 24fps as the first frame rate, 60fps as the second frame rate, frame A as the third image frame, and frame B as the fourth image frame as the fourth image frame as an example, the electronic device can determine that the ratio of the second frame rate to the first frame rate is 2.5. Since this ratio is not an integer, the second count is twice, the third count is three times, and the electronic device can read image frame A twice, image frame B three times, and so on, until the first layer of data is read.

[0078] For example, continue to refer to Figure 7Taking a frame rate of 30fps as the first frame rate, 60fps as the second frame rate, frame A as the third image frame, and frame B as the fourth image frame as the fourth image frame as an example, the electronic device can determine that the ratio of the second frame rate to the first frame rate is 2. Since this ratio is an integer, the first count is twice. The electronic device can read frame A twice, read frame B twice, and so on, until the first layer of data is read.

[0079] Based on the above scheme, since the number of times the first layer data is repeatedly read can be determined according to the ratio of the second frame rate to the first frame rate, the frame rate conversion from the first layer data to the third layer data can be realized, thus providing a basis for synthesizing the second layer data.

[0080] The image processing method provided in this application can be executed by an image processing device. This application uses an image processing device executing the image processing method as an example to illustrate the image processing device provided in this application.

[0081] like Figure 8 As shown in the illustration, this application also provides an image processing apparatus 800, including: an acquisition module 801, a processing module 802, and a display module 803; the acquisition module 801 is used to acquire first display data, wherein the first display data is composite data of multiple layers; the processing module 802 is used to perform a first processing on the first display data to obtain second display data, and perform a second processing on the second display data to obtain third display data; the display module 803 is used to display an image screen according to the third display data; wherein, the first processing includes frame dropping processing for odd-numbered or even-numbered frames, and the second processing includes inserting a third image frame between a first image frame and a second image frame, wherein the third image frame is an image frame determined based on the first image frame and the second image frame, and the first image frame and the second image frame are adjacent image frames in the second display data.

[0082] Optionally, the acquisition module 801 is further configured to acquire layer data to be displayed, the layer data to be displayed including first layer data; the processing module 802 is further configured to determine the first display data based on the first layer data and the second layer data when the layer data to be displayed also includes second layer data; and to perform frame interpolation processing on the first layer data to obtain fourth display data when the layer data to be displayed does not include the second layer data; the display module 803 is further configured to display an image screen according to the fourth display data.

[0083] Optionally, the processing module 802 is specifically configured to determine the display frame rate of the first layer data and the display frame rate of the second layer data, wherein the display frame rate of the first layer data is a first frame rate and the display frame rate of the second layer data is a second frame rate, and the first frame rate is less than the second frame rate; determine the number of repeated readings based on the first frame rate and the second frame rate, and read the first layer data according to the number of repeated readings to obtain the third layer data, wherein the display frame rate of the third layer data is the second frame rate; and perform layer merging processing on the second layer data and the third layer data to obtain the first display data.

[0084] Optionally, the processing module 802 is specifically configured to determine the ratio of the second frame rate to the first frame rate; if the ratio is an integer, repeatedly read each image frame of the first layer data according to the first number to obtain the third layer data, where the first number is the ratio; if the ratio is not an integer, repeatedly read the third image frame of the first layer data according to the second number, and repeatedly read the fourth image frame of the first layer data according to the third number to obtain the third layer data, where the second number is N, the third number is N+1, and the ratio is within the range of (N, N+1); wherein the third image frame and the fourth image frame are adjacent image frames in the first layer data, and N is a positive integer.

[0085] Optionally, the processing module 802 is specifically used to perform a first processing on the first display data through the system chip to obtain second display data; and to perform a second processing on the second display data through an independent display chip to obtain third display data.

[0086] In this embodiment, on the one hand, a third image frame can be inserted between the first image frame and the second image frame. Since the third image frame is determined based on the first image frame and the second image frame, inserting the third image frame can achieve a smooth transition between adjacent image frames, thereby improving the smoothness of the image. On the other hand, since the odd or even frames of the first display data can be dropped first and then the frame insertion can be performed, the processing burden of the second processing can be reduced, thereby reducing system power consumption and extending the device standby time.

[0087] The image processing device in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television set (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the device.

[0088] The image processing device in this application embodiment can be a device with an operating system. The operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system.

[0089] The image processing apparatus provided in this application embodiment can achieve... Figures 1 to 7 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.

[0090] Optionally, such as Figure 9 As shown, this application embodiment also provides an electronic device 900, including a processor 901 and a memory 902. The memory 902 stores a program or instructions that can run on the processor 901. When the program or instructions are executed by the processor 901, they implement the various steps of the above-described image processing method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0091] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0092] Figure 10 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application.

[0093] The electronic device 1000 includes, but is not limited to, components such as: radio frequency unit 1001, network module 1002, audio output unit 1003, input unit 1004, sensor 1005, display unit 1006, user input unit 1007, interface unit 1008, memory 1009, and processor 1010.

[0094] Those skilled in the art will understand that the electronic device 1000 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 1010 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 10 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0095] The processor 1010 is configured to acquire first display data, which is composite data of multiple layers; perform a first processing on the first display data to obtain second display data; and perform a second processing on the second display data to obtain third display data. The display unit 1006 is configured to display an image according to the third display data. The first processing includes frame dropping processing for odd or even frames, and the second processing includes inserting a third image frame between a first image frame and a second image frame. The third image frame is an image frame determined based on the first image frame and the second image frame, and the first image frame and the second image frame are adjacent image frames in the second display data.

[0096] In this embodiment, on the one hand, a third image frame can be inserted between the first image frame and the second image frame. Since the third image frame is determined based on the first image frame and the second image frame, inserting the third image frame can achieve a smooth transition between adjacent image frames, thereby improving the smoothness of the image. On the other hand, since the odd or even frames of the first display data can be dropped first and then the frame insertion can be performed, the processing burden of the second processing can be reduced, thereby reducing system power consumption and extending the device standby time.

[0097] Optionally, the processor 1010 is further configured to acquire layer data to be displayed, the layer data to be displayed including first layer data; if the layer data to be displayed also includes second layer data, determine first display data based on the first layer data and the second layer data; if the layer data to be displayed does not include the second layer data, perform frame interpolation processing on the first layer data to obtain fourth display data; the display unit 1006 is further configured to display an image screen according to the fourth display data.

[0098] In this embodiment, since the first display data can be determined based on the first layer data and the second layer data when the layer data to be displayed includes the first layer data and the second layer data, and the first layer data can be processed by frame interpolation to obtain the fourth display data when the layer data to be displayed does not include the second layer data, and the image is displayed according to the fourth display data, it is possible to achieve differentiated processing of the layer data to be displayed that includes the second layer data and the layer data that does not include the second layer data, thereby more effectively improving the image display effect of different video sources.

[0099] Optionally, the processor 1010 is specifically configured to determine the display frame rate of the first layer data and the display frame rate of the second layer data, wherein the display frame rate of the first layer data is a first frame rate and the display frame rate of the second layer data is a second frame rate, and the first frame rate is less than the second frame rate; determine the number of repeated reads based on the first frame rate and the second frame rate, and read the first layer data according to the number of repeated reads to obtain the third layer data, wherein the display frame rate of the third layer data is the second frame rate; and perform layer merging processing on the second layer data and the third layer data to obtain the first display data.

[0100] In this embodiment of the application, since the number of repeated readings can be determined based on the first frame rate and the second frame rate, and the third layer data can be obtained, a basis can be provided for merging the second layer data and the third layer data to obtain the first display data.

[0101] Optionally, the processor 1010 is specifically configured to determine the ratio of the second frame rate to the first frame rate; if the ratio is an integer, repeatedly read each image frame of the first layer data according to a first number to obtain the third layer data, wherein the first number is the ratio; if the ratio is not an integer, repeatedly read the third image frame of the first layer data according to a second number, and repeatedly read the fourth image frame of the first layer data according to a third number to obtain the third layer data, wherein the second number is N, the third number is N+1, and the ratio is within the range of (N, N+1); wherein the third image frame and the fourth image frame are adjacent image frames in the first layer data, and N is a positive integer.

[0102] In this embodiment, since the number of times the first layer data is repeatedly read can be determined based on the ratio of the second frame rate to the first frame rate, the frame rate conversion from the first layer data to the third layer data can be realized, thereby providing a basis for synthesizing the second layer data.

[0103] Optionally, the processor 1010 is specifically configured to perform a first processing on the first display data through a system chip to obtain second display data; and to perform a second processing on the second display data through a separate display chip to obtain third display data.

[0104] In this embodiment, since the first display data can be processed by the system chip to obtain the second display data, and the second display data can be processed by the independent display chip to obtain the third display data, the frame rate of the display data can be increased by the frame interpolation processing of the independent display chip while reducing the processing burden of the independent display chip, thereby improving the display effect of the image.

[0105] It should be understood that, in this embodiment, the input unit 1004 may include a graphics processing unit (GPU) 10041 and a microphone 10042. The GPU 10041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 1006 may include a display panel 10061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 1007 includes a touch panel 10071 and at least one of other input devices 10072. The touch panel 10071 is also called a touch screen. The touch panel 10071 may include a touch detection device and a touch controller. Other input devices 10072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be described in detail here.

[0106] The memory 1009 can be used to store software programs and various data. The memory 1009 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 1009 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 1009 in this embodiment includes, but is not limited to, these and any other suitable types of memory.

[0107] The processor 1010 may include one or more processing units; optionally, the processor 1010 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into the processor 1010.

[0108] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described image processing method embodiments and achieve the same technical effects. To avoid repetition, they will not be described again here.

[0109] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0110] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described image processing method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0111] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0112] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the above-described image processing method embodiments, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0113] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0114] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the related technology, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0115] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. An image processing method, characterized in that, include: The system chip acquires the data of the layer to be displayed, which includes the data of the first layer. If the layer data to be displayed also includes second layer data, the first display data is determined based on the first layer data and the second layer data. If the layer data to be displayed does not include the second layer data, the first layer data is subjected to frame interpolation to obtain the fourth display data, and the image is displayed according to the fourth display data. The first display data is obtained through the system chip, and the first display data is the composite data of multiple layers; The first display data is processed by the system chip to obtain the second display data, and the second display data is processed by the independent display chip to obtain the third display data. The image screen is displayed according to the third display data; The first processing includes frame dropping processing for odd or even frames, and the second processing includes inserting a third image frame between the first image frame and the second image frame. The third image frame is an image frame determined based on the first image frame and the second image frame, and the first image frame and the second image frame are adjacent image frames in the second display data.

2. The image processing method according to claim 1, characterized in that, Determining the first display data based on the first layer data and the second layer data includes: Determine the display frame rate of the first layer data and the display frame rate of the second layer data, wherein the display frame rate of the first layer data is a first frame rate and the display frame rate of the second layer data is a second frame rate, and the first frame rate is less than the second frame rate; The number of repeated reads is determined based on the first frame rate and the second frame rate, and the first layer data is read according to the number of repeated reads to obtain the third layer data, wherein the display frame rate of the third layer data is the second frame rate; The second layer data and the third layer data are merged to obtain the first display data.

3. The image processing method according to claim 2, characterized in that, The step of determining the number of repeated reads based on the first frame rate and the second frame rate, and reading the first layer data according to the number of repeated reads to obtain the third layer data, includes: Determine the ratio of the second frame rate to the first frame rate; When the ratio is an integer, each image frame of the first layer data is read repeatedly according to the first number to obtain the third layer data, where the first number is the ratio. When the ratio is not an integer, the third image frame of the first layer data is read repeatedly according to the second number, and the fourth image frame of the first layer data is read repeatedly according to the third number to obtain the third layer data. The second number is N, the third number is N+1, and the ratio is within the range of (N, N+1). Wherein, the third image frame and the fourth image frame are adjacent image frames in the first layer data, and N is a positive integer.

4. An image processing apparatus, characterized in that, include: Acquisition module, processing module, and display module; The acquisition module is used to acquire first display data, which is composite data of multiple layers; The processing module is used to perform a first processing on the first display data to obtain second display data, and to perform a second processing on the second display data to obtain third display data; The display module is used to display the image screen according to the third display data; The first processing includes frame dropping processing for odd or even frames; the second processing includes inserting a third image frame between the first image frame and the second image frame, wherein the third image frame is an image frame determined based on the first image frame and the second image frame, and the first image frame and the second image frame are adjacent image frames in the second display data; the acquisition module is further configured to acquire layer data to be displayed, wherein the layer data to be displayed includes first layer data; The processing module is further configured to, when the layer data to be displayed includes second layer data, determine the first display data based on the first layer data and the second layer data; and when the layer data to be displayed does not include the second layer data, perform frame interpolation on the first layer data to obtain fourth display data. The display module is also used to display the image screen according to the fourth display data.

5. The image processing apparatus according to claim 4, characterized in that, The processing module is specifically used to determine the display frame rate of the first layer data and the display frame rate of the second layer data, wherein the display frame rate of the first layer data is a first frame rate and the display frame rate of the second layer data is a second frame rate, and the first frame rate is less than the second frame rate; determine the number of repeated readings based on the first frame rate and the second frame rate, and read the first layer data according to the number of repeated readings to obtain the third layer data, wherein the display frame rate of the third layer data is the second frame rate; The second layer data and the third layer data are merged to obtain the first display data.

6. The image processing apparatus according to claim 5, characterized in that, The processing module is specifically used to determine the ratio of the second frame rate to the first frame rate; when the ratio is an integer, it repeatedly reads each image frame of the first layer data according to the first number of times to obtain the third layer data, wherein the first number of times is the ratio. When the ratio is not an integer, the third image frame of the first layer data is read repeatedly according to the second number, and the fourth image frame of the first layer data is read repeatedly according to the third number to obtain the third layer data. The second number is N, the third number is N+1, and the ratio is within the range of (N, N+1). The third image frame and the fourth image frame are adjacent image frames in the first layer data, and N is a positive integer.

7. An electronic device, characterized in that, This includes system-on-a-chip (SoC), discrete display chip (DIC), display, and data transmission interface; The system chip is used to acquire first display data, perform a first process on the first display data, and obtain second display data, wherein the first display data is composite data of multiple layers; The system chip is also used to acquire data of the layer to be displayed, the data of the layer to be displayed including data of the first layer; If the layer data to be displayed also includes second layer data, the first display data is determined based on the first layer data and the second layer data. If the layer data to be displayed does not include the second layer data, the first layer data is subjected to frame interpolation to obtain the fourth display data, and the image is displayed according to the fourth display data. The data transmission interface is used to transmit the second display data from the system chip to the independent display chip; The independent display chip is used to perform a second processing on the second display data to obtain a third display data; The display is used to display the image screen according to the third display data; The first processing includes frame dropping processing for odd or even frames, and the second processing includes inserting a third image frame between the first image frame and the second image frame. The third image frame is an image frame determined based on the first image frame and the second image frame, and the first image frame and the second image frame are adjacent image frames in the second display data.

8. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the image processing method as described in any one of claims 1-3.

Images