Image processing method and device, electronic equipment and readable storage medium

Abstract

The invention discloses an image processing method and device, electronic equipment and a readable storage medium, the method is suitable for a display chip, the display chip comprises a main body module and a frame insertion module, and the method comprises the following steps: the main body module receives first image data; the main body module performs super-resolution processing on the first image data to obtain second image data; wherein the resolution of the second image data is greater than that of the first image data; when the resolution of the second image data is greater than a target resolution threshold, the main body module cuts the second image data to obtain third image data; wherein the resolution of the third image data is smaller than or equal to the target resolution threshold, and the target resolution threshold is smaller than or equal to the maximum resolution supported by the frame insertion module; and the frame insertion module performs frame insertion processing according to the third image data to obtain target image data.

Description

Technical Field

[0001] This application belongs to the field of image processing, and specifically relates to an image processing method, apparatus, electronic device, and readable storage medium. Background Technology

[0002] Display chips, as dedicated image processing chips, have significant advantages in improving the display effects of electronic devices. Currently, the processing flow of a display chip is as follows: it receives image data from the system-on-a-chip (SoC), uses a super-resolution algorithm to perform super-resolution processing on the image data from the SoC, the super-resolution image data is further processed by a frame interpolation module, and finally the interpolated image data is transmitted to the display screen for display. In the super-resolution stage, the image data needs to be super-resolutiond to a higher resolution to obtain more image details and improve the display effect. However, the current frame interpolation module has a limited maximum resolution that it can support. If the resolution of the super-resolution image data exceeds the maximum resolution supported by the frame interpolation module, it may cause malfunction of the display chip, preventing normal display output, or even leading to screen freezing, screen distortion, or a black screen. In other words, current image processing methods cannot achieve higher image display effects while taking into account the current capabilities of display chips. Summary of the Invention

[0003] The purpose of this application is to provide an image processing method, apparatus, electronic device, and readable storage medium that can solve the problem that current image processing methods cannot achieve higher image display effects while taking into account the functions of current display chips.

[0004] In a first aspect, embodiments of this application provide an image processing method applicable to a display chip, wherein the display chip includes a main module and a frame interpolation module, and the main module is connected to the frame interpolation module;

[0005] The method includes:

[0006] The main module receives the first image data;

[0007] The main module performs super-resolution processing on the first image data to obtain second image data; wherein the resolution of the second image data is greater than the resolution of the first image data.

[0008] When the resolution of the second image data is greater than the target resolution threshold, the main module performs cropping processing on the second image data to obtain third image data; wherein the resolution of the third image data is less than or equal to the target resolution threshold, and the target resolution threshold is less than or equal to the maximum resolution supported by the frame interpolation module;

[0009] The frame interpolation module performs frame interpolation processing based on the third image data to obtain the target image data.

[0010] Secondly, embodiments of this application provide an image processing apparatus suitable for a display chip, the display chip including a main module and a frame interpolation module, the main module being connected to the frame interpolation module;

[0011] The main module includes:

[0012] A receiving unit is used to receive the first image data;

[0013] A super-resolution unit is used to perform super-resolution processing on the first image data to obtain second image data; wherein the resolution of the second image data is greater than the resolution of the first image data.

[0014] The cropping unit is used to crop the second image data when the resolution of the second image data is greater than the target resolution threshold to obtain the third image data; wherein the resolution of the third image data is less than or equal to the target resolution threshold, and the target resolution threshold is less than or equal to the maximum resolution supported by the frame interpolation module.

[0015] The frame interpolation module is used to perform frame interpolation processing based on the third image data to obtain the target image data.

[0016] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, wherein the memory stores a program or instructions executable on the processor, and the program or instructions, when executed by the processor, implement the steps of the image processing method as described in the first aspect.

[0017] Fourthly, embodiments of this application provide a readable storage medium storing a program or instructions that, when executed by a processor, implement the steps of the image processing method as described in the first aspect.

[0018] Fifthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the steps of the image processing method as described in the first aspect.

[0019] In a sixth aspect, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the steps of the image processing method as described in the first aspect.

[0020] In this embodiment, the main module in the display chip performs super-resolution processing on the received first image data to obtain second image data, making the resolution of the second image data greater than that of the first image data to increase image detail. If the resolution of the second image data is greater than a target resolution threshold, the main module performs cropping processing on the second image data to obtain third image data, making the resolution of the third image data less than or equal to the target resolution threshold. Since the target resolution threshold is less than or equal to the maximum resolution supported by the frame interpolation module, the frame interpolation process can be reliably executed, meaning that frame interpolation can be performed based on the third image data to obtain the target image. This scheme can balance the frame interpolation capability of the display chip with ensuring more image detail, thereby improving the image display effect. Attached Figure Description

[0021] Figure 1 This is one of the flowcharts of the image processing method according to an embodiment of this application;

[0022] Figure 2 This is a schematic diagram of the architecture of the display enhancement chip according to an embodiment of this application;

[0023] Figure 3 This is a second flowchart of the image processing method according to an embodiment of this application;

[0024] Figure 4 This is a block diagram of an image processing apparatus according to an embodiment of this application;

[0025] Figure 5 This is a block diagram of an electronic device according to an embodiment of this application;

[0026] Figure 6 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0027] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0028] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0029] The image processing method provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0030] like Figure 1 As shown, this application provides an image processing method applicable to a display chip, the display chip including: a main module and a frame interpolation module, the main module being connected to the frame interpolation module;

[0031] The method includes the following steps:

[0032] Step 11: The main module receives the first image data;

[0033] For example, the display chip can be disposed in an electronic device, which may further include a system-on-a-chip (SoC), and the main module of the display chip is connected to the SoC. Step 11 may include: the main module receiving first image data from the SoC. That is, the first image data may be image data transmitted from the SoC to the display chip.

[0034] Optionally, the first image data includes at least one of motion vector information, depth information, and color information. For example, the first image data may include motion vector and depth (MVD) information, and / or color information. The first image data may also be referred to as multimodal image data.

[0035] Step 12: The main module performs super-resolution processing on the first image data to obtain second image data; wherein the resolution of the second image data is greater than the resolution of the first image data;

[0036] In this embodiment, the second image data is the image data obtained by super-resolution processing of the first image data. For example, if the resolution of the second image data is the second resolution and the resolution of the first image data is the first resolution, then the second resolution is greater than the first resolution.

[0037] Optionally, the main module may include a super-resolution unit, which is connected to the frame interpolation module. Optionally, the display chip is disposed in the electronic device, and the super-resolution unit is connected to the system-on-a-chip (SoC) of the electronic device. The super-resolution unit can receive first image data from the SoC and perform super-resolution processing on the first image data to obtain second image data with a resolution higher than that of the first image data.

[0038] Optionally, the second image data includes at least one of motion vector information, depth information, and image information. For example, the second image data may include MVD information and / or color information. The second image data may also be referred to as multimodal image data.

[0039] Step 13: When the resolution of the second image data is greater than the target resolution threshold, the main module performs cropping processing on the second image data to obtain the third image data; wherein, the resolution of the third image data is less than or equal to the target resolution threshold, and the target resolution threshold is less than or equal to the maximum resolution supported by the frame interpolation module;

[0040] In this embodiment, the third image data is the image data obtained by cropping the second image data. For example, if the resolution of the second image data is the second resolution and the resolution of the third image data is the third resolution, then the third resolution is less than the second resolution. That is, the third image data with a resolution less than the second image data is obtained by cropping the second image data, so that the third resolution is less than or equal to the target resolution threshold, thereby ensuring that the frame interpolation module in the display chip can perform frame interpolation processing on the third image data.

[0041] Optionally, the display chip may further include a cropping unit, which is connected to both the super-resolution unit and the frame interpolation module. Optionally, the display chip is disposed in an electronic device, and the cropping unit can receive second image data from the super-resolution unit and perform cropping resolution processing on the second image data to obtain third image data with a resolution less than or equal to a target resolution threshold. In this way, super-resolution processing is performed on the first image data before cropping, ensuring that the frame interpolation module can perform frame interpolation processing while allowing the third image data to have more image details, thus improving image quality.

[0042] In this embodiment, the target resolution threshold is less than or equal to the maximum resolution supported by the frame interpolation process. That is, when the resolution of the second image data is greater than the target resolution threshold, the second image data is scaled to obtain a third image data with a resolution less than or equal to the maximum resolution supported by the frame interpolation process. This ensures that the frame interpolation process can be reliably executed, avoiding chip malfunctions, failure to display properly, or even screen freezing, screen distortion, or black screen.

[0043] Optionally, the third image data includes at least one of motion vector information, depth information, and image information. For example, the third image data may include MVD information and / or color information. The third image data may also be referred to as multimodal image data.

[0044] Step 14: The frame interpolation module performs frame interpolation processing based on the third image data to obtain the target image data.

[0045] In this embodiment, frame interpolation can be performed to increase the frame rate of the target image data.

[0046] It should be noted that the number of frames of the first image data in the embodiments of this application can be one frame or multiple frames of image data. That is, the above steps of the image processing method in the embodiments of this application can be processing for a certain frame of image data, or processing for multiple frames of images respectively, etc. The embodiments of this application are not limited thereto.

[0047] In the above scheme, the received first image data is super-resolution processed to obtain second image data, so that the resolution of the second image data is greater than that of the first image data, thereby increasing image detail. If the resolution of the second image data is greater than a target resolution threshold, the second image data is cropped to obtain third image data, so that the resolution of the third image data is less than or equal to the target resolution threshold. Since the target resolution threshold is less than or equal to the maximum resolution supported by the frame interpolation module, the frame interpolation process of the frame interpolation module can be reliably executed, i.e., frame interpolation can be performed based on the third image data to obtain the target image. This scheme can balance the frame interpolation capability of the display chip with ensuring more image detail, thereby improving the image display effect.

[0048] Furthermore, considering that super-resolution processing will generate high power consumption, in this embodiment of the application, super-resolution processing is performed first to retain more image details, and then frame interpolation is performed. This avoids increasing the number of images to be processed for super-resolution, thereby avoiding increased power consumption.

[0049] Optionally, the main module performs super-resolution processing on the first image data to obtain second image data, including:

[0050] Upon receiving multiple frames of first image data, the main module performs super-resolution processing on each frame of first image data to obtain multiple frames of second image data.

[0051] The multiple frames of second image data correspond one-to-one with the multiple frames of first image data.

[0052] For example, when the main module receives multiple frames of first image data from the system-on-a-chip, the main module can perform super-resolution processing on each frame of the multiple frames of first image data to obtain the second image data corresponding to each frame of first image data.

[0053] It should be noted that, in the embodiments of this application, after the super-resolution unit in the main module performs super-resolution processing on the first image data of the current frame and transmits the second image obtained by the super-resolution processing to the cropping unit or the frame interpolation unit, the super-resolution unit can then perform super-resolution processing on the first image data of the next frame without waiting for the cropping or frame interpolation processing to be completed, thereby improving the processing efficiency of the display chip.

[0054] Optionally, when the main module receives multiple frames of first image data, the method further includes:

[0055] If the resolution of the second image data of the target frame obtained after super-resolution processing of the first image data of the target frame is greater than the target resolution threshold, the main module stores the second image data in the memory as a reference frame for super-resolution processing of the next frame of the first image data of the target frame. The first image data of the target frame can be any frame of the first image data from the multiple frames.

[0056] In this embodiment, taking any frame of first image data, hereinafter referred to as the target frame first image data, as an example, since the resolution of the target frame second image data obtained after super-resolution processing of the target frame first image data is greater than the target resolution threshold, it is necessary to crop the target frame second image data to ensure that the frame interpolation module can reliably perform the frame interpolation process. Considering that the cropped third image data will lose some image information, in order to ensure that the target frame second image data can provide correct reference information for the super-resolution processing of the next frame of first image data of the target frame, this embodiment stores the target frame second image data before cropping the target frame second image data whose resolution is greater than the target resolution threshold. This allows the target frame second image data to be used as a reference frame for similar super-resolution processing during the super-resolution processing of the next frame of first image data of the target frame, thereby improving the accuracy of the super-resolution processing.

[0057] For example, the main module may further include the memory, which is connected to the super-resolution unit. The super-resolution unit can transmit the super-resolution processed second image data to the memory for storage. The super-resolution unit can also read the image data in the memory as a reference frame for super-resolution processing, etc., to ensure efficient and reliable execution of super-resolution processing. Alternatively, the memory may be set independently of the main module or independently of the display chip, etc., and the embodiments of this application are not limited thereto.

[0058] Optionally, the main module performs super-resolution processing on the first image data to obtain second image data, including:

[0059] According to a preset order, the information of each part of the second image data of the target frame is read from the memory in sequence;

[0060] Based on the information of each part of the second image data of the target frame read, super-resolution processing is performed on each part of the first image data of the next frame in sequence, and the information of each part after super-resolution processing is stored in the memory;

[0061] Based on all the super-resolution processed information stored in the memory, the second image data corresponding to the first image data of the next frame is obtained.

[0062] For example, a frame of second image data can be divided into multiple parts of information. For instance, a part of information may include one or more rows of pixel image data, or one or more columns of pixel image data, or one pixel block of image data, etc. This application's embodiments are not limited to these categories. For example, a pixel block may include N*M consecutive pixels, where N is less than the total number of rows in a frame of second image data, M is less than the total number of columns in a frame of second image data, and N and M are positive integers.

[0063] For example, if multiple parts of the second image data of the target frame are already stored in the memory, and super-resolution processing is needed for the first image data of the next frame of the target frame, the first part of the second image data of the target frame can be read from the memory first, the corresponding first part of the first image data of the next frame can be super-resolution processed, and the first part of the super-resolution processed first image data of the next frame can be stored in the memory. That is, the memory can store two frames of image data at this time. When the second image data of the second frame after the target frame is stored in the memory, the second image data of the target frame can be directly overwritten, thereby saving memory size and cost.

[0064] Furthermore, the second part of the target frame second image data is read from the memory, super-resolution processing is performed on the corresponding second part of the next frame first image data, and the super-resolution processed second part of the next frame first image data is stored in the memory. This process is repeated, that is, multiple parts of the stored target frame second image data can be read sequentially, and multiple parts of the next frame first image data can be super-resolution processed, and the super-resolution processed parts of the target frame first image data can be stored, thereby obtaining the next frame second image data.

[0065] It should be noted that each part of the information in the second image data corresponds one-to-one with each part of the information in the first image data. For example, if the first part of the information in the second image data is the first row of pixels, then the first part of the information in the first image data is also the first row of pixels, etc. This application embodiment is not limited to this.

[0066] In this embodiment, when performing super-resolution processing on the first image data of the target frame, the second image data after super-resolution processing is stored in the memory to provide reliable reference information for the super-resolution processing of the next frame's first image data, thereby improving the accuracy of the super-resolution processing. Furthermore, the memory only stores the target frame and the second image data for the next frame, avoiding read / write conflicts and saving memory size, thus reducing costs.

[0067] Optionally, when the main module receives multiple frames of first image data, the method further includes:

[0068] If the resolution of the second image data is less than or equal to the target resolution threshold, frame interpolation is performed in the frame interpolation module based on the second image data to obtain the target image data.

[0069] In this embodiment, when the resolution of the second image data is less than or equal to the target resolution threshold, that is, when the resolution of the second image data is less than or equal to the maximum resolution supported by the frame interpolation module, the frame interpolation process can be reliably executed. Therefore, the second image data with a resolution less than or equal to the target resolution threshold can be directly subjected to frame interpolation without going through the cropping unit, thereby improving image processing efficiency and reducing image transmission losses.

[0070] Optionally, when the main module receives multiple frames of first image data, the method further includes:

[0071] If the resolution of the second image data of the target frame obtained after super-resolution processing of the first image data of the target frame is less than or equal to the target resolution threshold, when super-resolution processing is performed on the first image data of the next frame of the first image data of the target frame, the second image data of the target frame is read from the frame interpolation module as the reference frame of the first image data of the next frame.

[0072] In this embodiment, considering that the interpolation module needs to calculate the intermediate frame image data based on two frames of image data when performing interpolation processing, if the resolution of the second image data is less than or equal to the target resolution threshold, it is not necessary to store the second image data whose resolution is less than or equal to the target resolution threshold. Instead, when performing super-resolution processing on the first image data of the target frame, the second image data of the previous frame of the target frame is selected as the reference frame from the two frames of image data used by the interpolation module when performing interpolation processing. This can save storage and reading processes and improve image processing efficiency.

[0073] Optionally, the display chip may further include a User Interface Blending (UIBlending) module, which is connected to the frame interpolation module and the system-on-a-chip (SoC). The UIBlending module is used to blend the target image data obtained after processing by the frame interpolation module with the user interface data received from the SoC to obtain image data to be displayed. Optionally, the UIBlending module can be connected to a display screen in an electronic device, and the image data to be displayed obtained by the UIBlending module can be transmitted to the display screen for image display.

[0074] In this embodiment, taking the image processing method executed by the display chip as an example, the display chip can significantly enhance the human eye's visual experience by processing the display frame rate, image resolution, and color adjustment. Simultaneously, in specific frame interpolation and super-resolution scenarios, it can also reduce the load on the system's central processing unit (CPU) and graphics processing unit (GPU), ensuring stable low-load operation of the system processor and thus achieving certain power consumption benefits.

[0075] like Figure 2 As shown in the figure, this application provides a schematic diagram of the architecture of a display chip; the display chip includes: a first receiving port Rx0, a second receiving port Rx1, an output port Tx0, a user interface mixing module, a main module, and a frame interpolation module, wherein the main module includes: a super-resolution unit, a memory, and a clipping unit. Here, the memory can also be called a storage module or a memory unit, etc., and the clipping unit can also be called a scaling unit, etc., and this application embodiment is not limited thereto.

[0076] It should be noted that the main module, including the super-resolution unit, memory, and pruning unit, can also be independent modules or units, and the embodiments of this application are not limited thereto.

[0077] The first receiving port Rx0 can be connected to the first serial display interface DSI0 of the System on Chip (SoC), and the second receiving port Rx1 can be connected to the second serial display interface DSI1 of the SoC. The first receiving port Rx0 is connected to the user interface mixing module, and the user interface mixing module is connected to the output port Tx0. For example, the first receiving port Rx0 can receive UI information from the SoC and transmit it directly to the user interface mixing module through path 1. The UI information transmitted through path 1 is the system resolution, such as 1440P. Of course, this embodiment is not limited to this.

[0078] Optionally, the display chip's receive ports Rx0 and Rx1 are used to receive the content to be displayed. A System-on-a-Chip (SoC) can consist of multiple integrated circuits with specific functions, such as, but not limited to, CPU, GPU, Digital Signal Processor (DSP), memory, and Modulator-Demodulator (Modem). The SoC's Display Serial Interface (DSI) is primarily used for communication between mobile devices (such as tablets and automotive displays) and the display panel. It utilizes differential signaling to transmit data, reducing power consumption while increasing data transmission rates.

[0079] In this system, the second receiving port Rx1 receives MVD and color information from the SoC, which is the first image data. This first image data is transmitted to the super-resolution unit via path 2, where it is super-resolutiond using a super-resolution algorithm. The super-resolution MVD and color information is then the second image data. This second image data can be transmitted to the memory via path 6 and to the cropping unit via path 3 to obtain scaled MVD and color information, which is the third image data. The third image data is then transmitted to the frame interpolation module via path 4; alternatively, the second image data can be directly transmitted to the frame interpolation module via path 7. The visual information after frame interpolation can be transmitted to the user interface mixing module via path 5.

[0080] Optionally, the cropping unit is used to crop the second image data when the output resolution of the super-resolution unit is greater than the upper limit of the processing capability of the frame interpolation module, so as to meet the processing requirements of the frame interpolation module.

[0081] The user interface hybrid module is connected to the display screen via output port Tx0.

[0082] Optionally, output port Tx0 is used to send the processed content. The user interface mixing module is used to synthesize the interpolated image data with UI information and output it to the display screen through output port Tx0 for display.

[0083] Optionally, the memory can be a random access memory (RAM) register used to store the MVD and color information of two frames of images. That is, when the super-resolution unit reads the MVD and color information from the RAM, it simultaneously stores the content of the image data of the frame being calculated into the memory to avoid read-write conflicts and abnormal situations. For example, if the memory already stores the target frame and the second frame of image data, when the second frame of image data arrives after the target frame, the target frame image data is directly overwritten, and so on. This can save memory size and reduce costs.

[0084] like Figure 3 As shown, this application embodiment also provides an image processing flow. Taking the maximum processing resolution of the frame interpolation module as 1080P, i.e., the target resolution threshold as 1080P, as an example, the specific flow includes:

[0085] Step 31: SoC input frame; for example: the first image data of the target frame input by the SoC;

[0086] Step 32: Super-resolution calculation; For example: The resolution of the first image data of the target frame input by the SoC is 720P. The first image data of the target frame is super-resolution calculated in the super-resolution unit, and the second image data after super-resolution magnification is output.

[0087] Step 33: Determine whether the resolution of the super-amplified second image data output by the super-resolution unit is greater than the processing capacity limit of the frame interpolation module, i.e., whether it is greater than the maximum resolution supported by the frame interpolation module. For example: if the resolution of the super-amplified second image data output by the super-resolution unit is 1440P, and 1440P is greater than 1080P, which exceeds the processing resolution limit of the frame interpolation module, then execute steps 34 and 35; as another example: if the resolution of the super-amplified second image data is 1080P, and 1080P is within the processing capacity range of the frame interpolation module, then execute step 36.

[0088] Step 34: Perform cropping processing; for example, the cropping unit can crop the 1440P second image data, such as cropping it into a 1080P or smaller image, thereby meeting the calculation requirements of the frame interpolation module.

[0089] Step 35: Store the super-resolution magnified image; for example, the cropped second image data will not provide the super-resolution unit with the correct reference information for super-resolution calculation of the next frame of the first image data, which will lead to inaccurate calculations or even anomalies in the super-resolution unit. Therefore, by storing the 1440P second image data processed by the super-resolution unit in memory, the super-resolution unit can directly read the complete 1440P second image data from memory when performing super-resolution calculations on the next frame of the first image data, thus correctly super-resolution the next frame of the first image data to 1440P and avoiding abnormal display situations.

[0090] Step 36: Perform frame interpolation calculation; for example, the frame interpolation module can perform frame interpolation on the cropped third image data. Alternatively, when it is determined that the resolution of the super-resolution amplified second image data (e.g., 1080P) output by the super-resolution unit is less than or equal to the upper limit of the frame interpolation module's processing capability, frame interpolation can be directly performed on the 1080P second image data output by the super-resolution unit. In this case, the cropping unit does not perform cropping on the second image data. Considering the internal algorithm characteristics of the frame interpolation module, the content of two frames is needed to calculate the intermediate frame. Therefore, the super-resolution unit can refer to the cropped and super-resolution image information of the previous second image to perform similar super-resolution processing on the cropped first image of the next frame. In this case, it is not necessary to perform image data storage operations in the internal memory.

[0091] Step 37: Output to display screen.

[0092] Specifically, frame interpolation involves motion estimation and motion compensation. For example, the frame interpolation process can be divided into two stages: motion estimation and motion compensation. In the motion estimation stage: a frame is divided into M*N non-overlapping pixel blocks of uniform size, where all pixels within each block have the same displacement. A matching pixel block is searched for among all pixel blocks according to a certain search principle. The relative displacement between the matching pixel block and the current pixel block is calculated, yielding the motion vector. In the motion compensation stage: adjacent frames are calculated as linear motions, meaning that in terms of temporal relationship, the intermediate frame lies on the linear motion trajectory of adjacent frames. Operations on all pixel blocks yield the interpolated frame. Furthermore, considering that in practical applications, there may be situations where matching blocks cannot be accurately found, the degree of pixel block matching can be used to indicate the similarity. For example, the Sum of Absolute Differences (SAD) can be used to represent the similarity. A smaller SAD indicates a more reliable motion vector; a larger SAD indicates a less reliable motion vector. During frame interpolation, SAD can be generated in real time frame by frame to determine the reliability of motion vectors. If the motion vectors are reliable based on SAD, the motion compensation result will meet expectations. If the motion vectors are unreliable based on SAD, corresponding strategies can be used for weighted fusion or the previous frame operation can be repeated to ensure that the motion compensation result meets expectations.

[0093] Specifically, super-resolution processing is the process of recovering image details and other data information based on known image information using optics and related optical knowledge. In other words, it increases the image resolution and prevents image quality degradation. For example, nearest neighbor interpolation or other methods can be used for super-resolution processing, but the embodiments in this application are not limited to these.

[0094] In this embodiment, a cropping unit and a memory are added inside the display chip to enhance its performance. The memory can store two frames of image data. When the super-resolution unit performs super-resolution calculations, it can obtain complete image data from the memory, thus ensuring the accuracy of the calculations. The cropping unit can crop the image when the super-resolution unit outputs image data with a resolution greater than the maximum processing resolution of the frame interpolation module. This ensures the normal processing function of the entire frame interpolation module, guarantees image quality, and avoids severe fragmentation, black screens, screen tearing, and freezing, thereby improving the user's viewing experience.

[0095] The image processing method provided in this application can be executed by an image processing device. This application uses an image processing device to perform the image processing method as an example to illustrate the image processing device provided in this application.

[0096] like Figure 4 As shown, this application provides an image processing device 400 suitable for a display chip. The display chip includes a main module 410 and a frame interpolation module 420, wherein the main module 410 is connected to the frame interpolation module 420.

[0097] The main module 410 includes:

[0098] The receiving unit 4110 is used to receive the first image data;

[0099] The super-resolution unit 4120 is used to perform super-resolution processing on the first image data to obtain second image data; wherein the resolution of the second image data is greater than the resolution of the first image data.

[0100] The cropping unit 4130 is used to crop the second image data to obtain third image data when the resolution of the second image data is greater than the target resolution threshold; wherein the resolution of the third image data is less than or equal to the target resolution threshold, and the target resolution threshold is less than or equal to the maximum resolution supported by the frame interpolation module.

[0101] The frame interpolation module 420 is used to perform frame interpolation processing based on the third image data to obtain the target image data.

[0102] Optionally, the super-resolution unit 4120 includes:

[0103] The first supramolecular unit is used to perform super-resolution processing on each frame of the first image data in the received multi-frame first image data to obtain multi-frame second image data.

[0104] The multiple frames of second image data correspond one-to-one with the multiple frames of first image data.

[0105] Optionally, when the receiving unit 4110 receives multiple frames of first image data, the main module 410 further includes:

[0106] Storage module 4140 is used to store the target frame second image data in a memory when the resolution of the target frame second image data obtained after super-resolution processing of the target frame first image data is greater than the target resolution threshold, and use it as a reference frame when performing super-resolution processing on the next frame first image data of the target frame first image data.

[0107] Wherein, the target frame first image data is any frame first image data among the multiple frame first image data.

[0108] Optionally, the super-resolution unit 4120 includes:

[0109] The first reading subunit is used to read information from the memory of each part of the second image data of the target frame in a preset order.

[0110] The second supramolecular unit is used to perform super-resolution processing on each part of the information of the second image data of the next frame in sequence according to the information of each part of the second image data of the target frame read, and to store the information of each part after super-resolution processing into the memory.

[0111] The storage subunit is used to obtain the second image data corresponding to the first image data of the next frame based on all the super-resolution processed information stored in the memory.

[0112] Optionally, when the receiving unit 4110 receives multiple frames of first image data, the main module 410 further includes:

[0113] The second reading subunit is used to read the target frame second image data as a reference frame for the next frame first image data when performing super-resolution processing on the target frame first image data, provided that the resolution of the target frame second image data obtained after super-resolution processing on the target frame first image data is less than or equal to the target resolution threshold.

[0114] Optionally, the frame interpolation module 420 is further configured to perform frame interpolation processing on the second image data to obtain target image data when the resolution of the second image data is less than or equal to the target resolution threshold.

[0115] The apparatus in this embodiment performs super-resolution processing on received first image data to obtain second image data, such that the resolution of the second image data is greater than that of the first image data, thereby increasing image detail. If the resolution of the second image data is greater than a target resolution threshold, the second image data is cropped to obtain third image data, such that the resolution of the third image data is less than or equal to the target resolution threshold. Since the target resolution threshold is less than or equal to the maximum resolution supported by the frame interpolation module, the frame interpolation process of the frame interpolation module can be reliably executed, meaning that frame interpolation can be performed based on the third image data to obtain the target image. This scheme can balance the frame interpolation capability of the display chip with ensuring more image detail, thereby improving the image display effect.

[0116] The image processing device in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television set (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the device.

[0117] The image processing device in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system.

[0118] The image processing apparatus provided in this application embodiment can achieve... Figures 1 to 3 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.

[0119] Optionally, such as Figure 5As shown, this application embodiment also provides an electronic device 500, including a processor 501 and a memory 502. The memory 502 stores a program or instructions that can run on the processor 501. When the program or instructions are executed by the processor 501, they implement the various steps of the above-described image processing method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0120] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0121] Figure 6 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application.

[0122] The electronic device 600 includes, but is not limited to, components such as: radio frequency unit 601, network module 602, audio output unit 603, input unit 604, sensor 605, display unit 606, user input unit 607, interface unit 608, memory 609, and processor 610.

[0123] Those skilled in the art will understand that the electronic device 600 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 610 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 6 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0124] The electronic device further includes a display chip, which comprises a main module and a frame interpolation module, the main module being connected to the frame interpolation module; the main module is used for:

[0125] Receive the first image data;

[0126] The first image data is subjected to super-resolution processing to obtain second image data; wherein the resolution of the second image data is greater than the resolution of the first image data.

[0127] If the resolution of the second image data is greater than the target resolution threshold, the second image data is cropped to obtain the third image data; wherein the resolution of the third image data is less than or equal to the target resolution threshold, and the target resolution threshold is less than or equal to the maximum resolution supported by the frame interpolation module.

[0128] The third image data is used for frame interpolation in the frame interpolation module to obtain the target image data.

[0129] Optionally, the main module is further configured to:

[0130] Upon receiving multiple frames of first image data, super-resolution processing is performed on each frame of the first image data to obtain multiple frames of second image data.

[0131] The multiple frames of second image data correspond one-to-one with the multiple frames of first image data.

[0132] Optionally, the main module is further configured to:

[0133] If the resolution of the second image data of the target frame obtained after super-resolution processing of the first image data of the target frame is greater than the target resolution threshold, the second image data of the target frame is stored in the memory and used as a reference frame when super-resolution processing is performed on the next frame of the first image data of the target frame.

[0134] Wherein, the target frame first image data is any frame first image data among the multiple frame first image data.

[0135] Optionally, the main module is further configured to:

[0136] According to a preset order, the information of each part of the second image data of the target frame is read from the memory in sequence;

[0137] Based on the information of each part of the second image data of the target frame read, super-resolution processing is performed on each part of the first image data of the next frame in sequence, and the information of each part after super-resolution processing is stored in the memory;

[0138] Based on all the super-resolution processed information stored in the memory, the second image data corresponding to the first image data of the next frame is obtained.

[0139] Optionally, the main module is further configured to:

[0140] If the resolution of the second image data of the target frame obtained after super-resolution processing of the first image data of the target frame is less than or equal to the target resolution threshold, when super-resolution processing is performed on the first image data of the next frame of the first image data of the target frame, the second image data of the target frame is read from the frame interpolation module as the reference frame of the first image data of the next frame.

[0141] Optionally, the main module is further configured to:

[0142] If the resolution of the second image data is less than or equal to the target resolution threshold, frame interpolation is performed in the frame interpolation module based on the second image data to obtain the target image data.

[0143] The electronic device in this embodiment performs super-resolution processing on received first image data to obtain second image data, such that the resolution of the second image data is greater than that of the first image data, thereby increasing image detail. If the resolution of the second image data is greater than a target resolution threshold, the second image data is cropped to obtain third image data, such that the resolution of the third image data is less than or equal to the target resolution threshold. Since the target resolution threshold is less than or equal to the maximum resolution supported by the frame interpolation module, the frame interpolation process of the frame interpolation module can be reliably executed, meaning that frame interpolation can be performed based on the third image data to obtain the target image. This scheme can balance the frame interpolation capability of the display chip with ensuring more image detail, thereby improving the image display effect.

[0144] It should be understood that, in this embodiment, the input unit 604 may include a graphics processing unit (GPU) 6041 and a microphone 6042. The GPU 6041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 606 may include a display panel 6061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 607 includes at least one of a touch panel 6071 and other input devices 6072. The touch panel 6071 is also called a touch screen. The touch panel 6071 may include a touch detection device and a touch controller. Other input devices 6072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be described in detail here.

[0145] The memory 609 can be used to store software programs and various data. The memory 609 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 609 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 609 in this embodiment includes, but is not limited to, these and any other suitable types of memory.

[0146] Processor 610 may include one or more processing units; optionally, processor 610 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 610.

[0147] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described image processing method embodiments and achieve the same technical effects. To avoid repetition, they will not be described again here.

[0148] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0149] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described image processing method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0150] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0151] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the above-described image processing method embodiments, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0152] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0153] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0154] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. An image processing method, characterized in that, This is applicable to display chips, which include a main module and a frame interpolation module, with the main module connected to the frame interpolation module. The method includes: The main module receives the first image data; The main module performs super-resolution processing on the first image data to obtain second image data; wherein the resolution of the second image data is greater than the resolution of the first image data. When the resolution of the second image data is greater than the target resolution threshold, the main module performs cropping processing on the second image data to obtain third image data; wherein the resolution of the third image data is less than or equal to the target resolution threshold, and the target resolution threshold is less than or equal to the maximum resolution supported by the frame interpolation module; The frame interpolation module performs frame interpolation processing based on the third image data to obtain the target image data.

2. The image processing method according to claim 1, characterized in that, The main module performs super-resolution processing on the first image data to obtain second image data, including: Upon receiving multiple frames of first image data, the main module performs super-resolution processing on each frame of the first image data to obtain multiple frames of second image data. The multiple frames of second image data correspond one-to-one with the multiple frames of first image data.

3. The image processing method according to claim 1 or 2, characterized in that, When the main module receives multiple frames of first image data, the method further includes: When the resolution of the second image data of the target frame obtained after super-resolution processing of the first image data of the target frame is greater than the target resolution threshold, the main module stores the second image data of the target frame in the memory as a reference frame when performing super-resolution processing on the next frame of the first image data of the target frame. Wherein, the target frame first image data is any frame first image data among the multiple frame first image data.

4. The image processing method according to claim 3, characterized in that, The main module performs super-resolution processing on the first image data to obtain second image data, including: According to a preset order, the information of each part of the second image data of the target frame is read from the memory in sequence; Based on the information of each part of the second image data of the target frame read, super-resolution processing is performed on each part of the first image data of the next frame in sequence, and the information of each part after super-resolution processing is stored in the memory; Based on all the super-resolution processed information stored in the memory, the second image data corresponding to the first image data of the next frame is obtained.

5. The image processing method according to claim 1 or 2, characterized in that, When the main module receives multiple frames of first image data, the method further includes: When the resolution of the second image data of the target frame obtained after super-resolution processing of the first image data of the target frame is less than or equal to the target resolution threshold, the main module reads the second image data of the target frame from the frame interpolation module as the reference frame of the next frame of the first image data when performing super-resolution processing on the first image data of the next frame of the target frame.

6. The image processing method according to claim 1, characterized in that, After the main module performs super-resolution processing on the first image data to obtain the second image data, it further includes: When the resolution of the second image data is less than or equal to the target resolution threshold, the main module performs frame interpolation processing on the frame interpolation module based on the second image data to obtain the target image data.

7. An image processing apparatus, characterized in that, This is applicable to display chips, which include a main module and a frame interpolation module, with the main module connected to the frame interpolation module. The main module includes: A receiving unit is used to receive the first image data; A super-resolution unit is used to perform super-resolution processing on the first image data to obtain second image data; wherein the resolution of the second image data is greater than the resolution of the first image data. The cropping unit is used to crop the second image data when the resolution of the second image data is greater than the target resolution threshold to obtain the third image data; wherein the resolution of the third image data is less than or equal to the target resolution threshold, and the target resolution threshold is less than or equal to the maximum resolution supported by the frame interpolation module. The frame interpolation module is used to perform frame interpolation processing based on the third image data to obtain the target image data.

8. The image processing apparatus according to claim 7, characterized in that, When the receiving unit receives multiple frames of first image data, the main module further includes: The storage unit is used to store the target frame second image data in the memory when the resolution of the target frame second image data obtained after super-resolution processing of the target frame first image data is greater than the target resolution threshold, and to serve as a reference frame when super-resolution processing is performed on the next frame of the target frame first image data. Wherein, the target frame first image data is any frame first image data among the multiple frame first image data.

9. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a program or instructions that can run on the processor, the program or instructions being executed by the processor to implement the steps of the image processing method as described in any one of claims 1 to 6.

10. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the image processing method as described in any one of claims 1 to 6.

Images