Image processing method, apparatus and electronic device

Abstract

The application provides an image processing method, device and electronic equipment. After an image enhancement processing device obtains a to-be-displayed image, the image enhancement processing device performs image enhancement processing on the to-be-displayed image that meets an image enhancement condition, such as a video image, a game image and the like, to ensure the quality of an output target display image and meet the image processing requirement for an image with low quality. The image enhancement processing is not performed on the to-be-displayed image that does not meet the image enhancement condition, such as a desktop image, a text document interface image and the like, and the to-be-displayed image is directly output as the target display image, thereby solving the technical problem that indiscriminate enhancement processing is performed on all rendered images, additional interference data is introduced in the enhancement processing of the images, and the quality of the output image is reduced.

Description

Technical Field

[0001] This application relates primarily to the field of computer applications, and more specifically to an image processing method, apparatus, and electronic device. Background Technology

[0002] With the rapid development of electronic device screens, high frame rates and high refresh rates have become a major trend in screen display requirements. Therefore, based on the high-performance graphics cards configured in electronic devices, this paper proposes to configure an image enhancement processing chip between the electronic device's image processor and the display screen. This chip performs image enhancement processing on each frame of the rendered image before sending the enhanced image to the display screen for output, thereby improving image display quality.

[0003] However, this image enhancement method consumes a lot of computing resources, reducing image processing efficiency; and for the display content of certain applications on the screen, such as desktop images and text documents, additional information is often introduced during the pixel or encoding / decoding enhancement process, reducing the clarity and accuracy of the output image. Summary of the Invention

[0004] In view of this, this application proposes an image processing method, the method comprising:

[0005] Obtain the image to be displayed;

[0006] Once it is determined that the image to be displayed meets the image enhancement conditions, the image to be displayed is enhanced to obtain the target display image;

[0007] If it is determined that the image to be displayed does not meet the image enhancement conditions, the image to be displayed is determined as the target display image;

[0008] Output the target display image.

[0009] Optionally, obtaining the image to be displayed includes:

[0010] The image to be displayed is rendered and output by the receiving and processing device;

[0011] The output of the target display image includes:

[0012] The target display image is sent to the display device for display.

[0013] or,

[0014] Obtaining the image to be displayed includes:

[0015] Acquire the image to be displayed after being rendered by the processing device;

[0016] The output of the target display image includes:

[0017] The target display image is fed back to the processing device, which then sends the target display image to the display device for display.

[0018] Optionally, determining that the image to be displayed satisfies the image enhancement conditions includes:

[0019] Obtain the scene type to which the image to be displayed belongs;

[0020] If the scene type belongs to the first type of scene, the image to be displayed is determined to meet the image enhancement conditions;

[0021] The image to be displayed does not meet the image enhancement conditions in the following ways:

[0022] If the scene type belongs to the second type of scene, it is determined that the image to be displayed does not meet the image enhancement conditions.

[0023] Optionally, the enhancement processing of the image to be displayed to obtain the target display image includes:

[0024] Identify the image region to be enhanced contained in the image to be displayed;

[0025] The image region to be enhanced contained in the image to be displayed is subjected to enhancement processing to obtain the target display image.

[0026] Optionally, identifying the image region to be enhanced contained in the image to be displayed includes:

[0027] Based on an attention mechanism, the region of interest (ROI) contained in the image to be displayed is determined as the region of image to be enhanced; the ROI includes the first application content in the first type of scenario; or,

[0028] Determining that the content of the image to be displayed belongs to the first application content, the entire display area of ​​the image to be displayed is determined as the image area to be enhanced; and / or,

[0029] The image to be displayed is determined to include first application content and second application content, and the area where the first application content is located is identified as the area of ​​the image to be enhanced.

[0030] Optionally, the method further includes:

[0031] The obtained image to be displayed is written into the processing queue to determine frame by frame whether the cached image to be displayed meets the image enhancement conditions; and / or,

[0032] The obtained target display image is written to the output queue to output the cached target display image frame by frame; and / or,

[0033] The enhanced image to be displayed is then calibrated to obtain the target display image.

[0034] This application also proposes an image processing apparatus, the apparatus comprising:

[0035] The image to be displayed module is used to obtain the image to be displayed.

[0036] The first determining module is used to determine that the image to be displayed meets the image enhancement conditions, and to perform enhancement processing on the image to be displayed to obtain the target display image;

[0037] The second determining module is used to determine that the image to be displayed does not meet the image enhancement conditions, and to determine the image to be displayed as the target display image;

[0038] The target display image output module is used to output the target display image.

[0039] This application also proposes an electronic device, the electronic device comprising:

[0040] A processing unit for performing image rendering operations to obtain an image to be displayed;

[0041] An image enhancement processing apparatus for implementing the image processing method described above;

[0042] A display device is used to display the image enhancement processing device or the target display image output by the processing device.

[0043] Optionally, the image enhancement processing device includes:

[0044] A data transmission port is used to connect to the processing device and receive the image to be displayed output by the processing device;

[0045] An image enhancement processor is configured to connect to the data transmission port, receive the image to be displayed transmitted by the data transmission port, determine that the image to be displayed meets the image enhancement conditions, perform enhancement processing on the image to be displayed to obtain a target display image; determine that the image to be displayed does not meet the image enhancement conditions, determine the image to be displayed as the target display image, and send the target display image to the data transmission port.

[0046] The data transmission port is also used to send the target display image to the processing device, which then sends the target display image to the connected display device; or it is also used to connect to the display device and send the target display image to the display device.

[0047] Optionally, the image enhancement processor includes:

[0048] An image preprocessing unit is configured to connect to the data transmission port, receive the image to be displayed transmitted by the data transmission port, determine whether the image to be displayed meets the image enhancement conditions, determine the image to be displayed that does not meet the image enhancement conditions as the target display image, and send the target display image to the data transmission port.

[0049] An image enhancement processing unit is used to connect the image preprocessing unit and the data transmission port, enhance the image to be displayed that meets the image enhancement conditions, obtain a target display image, and send the target display image to the data transmission port;

[0050] The data transmission port includes a first data transmission port with bidirectional data transmission function. The first data transmission port is connected to the processing device to receive the image to be displayed output by the processing device and send the received target display image to the processing device.

[0051] Alternatively, the data transmission port may include a second data transmission port with unidirectional data transmission function, which is connected to the processing device and the display device respectively, to receive the image to be displayed output by the processing device and send the received target display image to the display device.

[0052] Therefore, this application provides an image processing method, apparatus, and electronic device. After obtaining the image to be displayed, the image enhancement processing apparatus will only perform image enhancement processing on images that meet the image enhancement conditions, such as video images and game images, to ensure the quality of the output image. For images that do not meet the image enhancement conditions, such as desktop images and text document interface images, image enhancement processing can be skipped and the images can be directly output as the target display images. This solves the technical problem that performing indiscriminate enhancement processing on all rendered images leads to the introduction of additional interference data and reduces the quality of the output image. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0054] Figure 1 A schematic diagram of the hardware structure of an optional example of an electronic device suitable for the image processing method proposed in this application;

[0055] Figure 2A schematic diagram of the hardware structure of another optional example of an electronic device suitable for the image processing method proposed in this application;

[0056] Figure 3a A schematic diagram of the hardware structure of another optional example of an electronic device suitable for the image processing method proposed in this application;

[0057] Figure 3b A schematic diagram of the hardware structure of another optional example of an electronic device suitable for the image processing method proposed in this application;

[0058] Figure 4 A schematic diagram of the hardware structure of another optional example of an electronic device suitable for the image processing method proposed in this application;

[0059] Figure 5a A schematic diagram of the hardware structure of another optional example of an electronic device suitable for the image processing method proposed in this application;

[0060] Figure 5b A schematic diagram of the hardware structure of another optional example of an electronic device suitable for the image processing method proposed in this application;

[0061] Figure 5c A schematic diagram of the hardware structure of another optional example of an electronic device suitable for the image processing method proposed in this application;

[0062] Figure 6a A schematic diagram of the hardware structure of another optional example of an electronic device suitable for the image processing method proposed in this application;

[0063] Figure 6b A schematic diagram of the hardware structure of another optional example of an electronic device suitable for the image processing method proposed in this application;

[0064] Figure 7 A schematic diagram of the driving architecture of an electronic device applicable to the image processing method proposed in this application;

[0065] Figure 8 A schematic flowchart illustrating an optional example of the image processing method proposed in this application;

[0066] Figure 9 This is a flowchart illustrating yet another optional example of the image processing method proposed in this application.

[0067] Figure 10 This is a flowchart illustrating yet another optional example of the image processing method proposed in this application.

[0068] Figure 11a The image to be displayed is for use with the image processing method proposed in this application;

[0069] Figure 11b A schematic diagram of the detection results of the region to be enhanced in the image to be displayed, applicable to the image processing method proposed in this application;

[0070] Figure 12 This is a control timing diagram applicable to the image processing method proposed in this application;

[0071] Figure 13 This is a schematic diagram of an optional example of the image processing apparatus proposed in this application. Detailed Implementation

[0072] Regarding the description in the background section, this application proposes to selectively perform enhancement processing on images to be displayed after image rendering operations are performed by an electronic device processing apparatus, and then send the processed image to the display for output, thereby improving image display quality and meeting application display requirements. Specifically, since images to be displayed that do not require enhancement processing can be directly sent to the display for display after rendering, this not only reduces computational resource consumption but also avoids enhancing such images, which would introduce additional information and degrade the display image quality.

[0073] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0074] Reference Figure 1 This is a schematic diagram of the hardware structure of an optional example of an electronic device applicable to the image processing method proposed in this application. The electronic device may include, but is not limited to, smartphones, tablets, wearable devices, augmented reality (AR) devices, virtual reality (VR) devices, in-vehicle devices, smart medical devices, smart transportation devices, desktop computers, and other terminals. The type of electronic device can be determined based on the image processing application environment. Figure 1 As shown, the electronic device may include, but is not limited to, a processing device 100, an image enhancement processing device 200, and a display device 300, wherein:

[0075] The processing device 100 can be used to perform image rendering operations to obtain an image to be displayed. The composition and structure of the processing device 100 can be determined based on the working principle of image rendering, and this application does not limit the hardware structure of the processing device 100.

[0076] Optional, such as Figure 2As shown, the processing device 100 may include a graphics processing unit (GPU) 110, a main processor 120 (such as a central processing unit (CPU)) connected to the GPU 110, and a display selector 130 connected to the GPU 110 and the main processor 120 respectively.

[0077] In practical applications, the graphics processor 110 and the main processor 120 can cooperate to perform image rendering processing. This process can adjust the communication state of the display selector 130 and change the communication channels between the components within the processing device 100. This application does not describe the image rendering process in detail. During the display control process of the electronic device, the display selector 130 can be controlled to connect to the graphics processor 110 or the main processor 120 to realize the transmission of the image rendered by the graphics processor 110 or the transmission of control commands / signals output by the main processor 120, thereby meeting different application control requirements. This application does not limit the implementation process.

[0078] Among them, such as Figure 2 As shown, the graphics processor 110 and the main processor 120 can be connected via a PEG (PCI Express) interface, and these two processors can be connected via an eDP (embedded DisplayPort) unidirectional communication interface to an eDP switch (i.e., display selector 130). Thus, in image processing application scenarios, the image to be displayed rendered by the graphics processor 110 and the main processor 120, or the enhanced target display image forwarded, can be sent to the display selector 130 via eDP. The principle of eDP communication is not described in detail in this application.

[0079] Optionally, the aforementioned processing device 100 may also include an integrated southbridge chip PCH (Platform Controller Hub), which can be connected to the main processor 120 via a direct media interface (DMI) to meet the system control requirements of the electronic device. This application does not limit the application functions of PCH, CPU, and GPU in electronic devices, and can be determined as appropriate.

[0080] The image enhancement processing apparatus 200 can be used to implement the image processing method proposed in this application. In the embodiments of this application, the image enhancement processing apparatus 200 may be an enhancement processing chip, containing functional modules that implement different steps of the image processing method. This application does not limit its composition and structure.

[0081] The display device 300 can be used to display the target display image output by the image enhancement processing device 200 or the processing device 100. This can be determined based on the communication connection between the image enhancement processing device 200, the processing device 100, and the display device 300. This application does not limit the display principle of the target display image, and it can be determined based on the display method of the display device 300 (such as discrete graphics or integrated graphics output, etc.). This embodiment will not be described in detail here.

[0082] In some embodiments, the image enhancement processing device 200 may be connected to the processing device 100 and the display device 300 respectively, such as Figure 3a As shown, the image enhancement processing device 200 can be connected to the eDP Switch (i.e., display selector 130) via the eDP interface (here referring to the eDP uplink RX interface), and then connected to the display device 300 via the eDP downlink interface. In this way, the image enhancement processing device 200 can receive a frame of image to be displayed output by the processing device 100 after rendering. If it is determined that the image to be displayed meets the image enhancement conditions (i.e., the conditions used to indicate that the image to be displayed needs to be enhanced, the content of which is not limited in this application, but can be determined as appropriate, and the implementation process can be referred to the description of the corresponding part of the method embodiment below), the image to be displayed is enhanced, and the obtained target display image is sent to the display device 300 for display.

[0083] In some other embodiments, unlike the eDP unidirectional communication method described above, this application proposes to adopt a bidirectional communication method to realize the communication connection between the image enhancement processing device 200 and the processing device 100, so as to leverage the parallel processing capability of the graphics processing unit (GPU) in the processing device 200. Based on this, as Figure 3b The schematic diagram shows another alternative example of the hardware structure of the electronic device. The image enhancement processing device 200 can be connected to the processing device 100 via a bidirectional communication interface, such as... Figure 3b As shown, the image enhancement processing device 200 can be connected to the main processor CPU of the processing device 100 via a bidirectional communication interface (this application does not limit the type of result or its working principle, and can be configured according to communication requirements), and the eDP Switch of the processing device 100 can be connected to the display device 300 via the eDP interface.

[0084] In this case, the processing device 100 performs an image rendering operation to obtain an image to be displayed. Before outputting the image to be displayed to the display device 300, the image to be displayed can be sent to the image enhancement processing device 200 through a bidirectional communication interface, so that the image enhancement processing device 200 can obtain the target display image to be sent to the display device 300, such as the image to be displayed or its enhanced image, according to the image processing method proposed in this application. Then, the target display image is fed back to the CPU of the processing device 100 through the bidirectional communication interface, and then sent to the display device 300 for display through the eDP unidirectional transmission channel.

[0085] As can be seen, compared to the above... Figure 3a The illustrated electronic device system architecture uses a one-way communication method. Figure 3b The system architecture shown employs bidirectional communication, enabling data interaction between the image enhancement processing device 200 and the processing device 100. This better meets image processing requirements and allows the image enhancement processing device 200 to extract the rendered image containing a complete frame of image content or partial image content from the GPU of the processing device 100, without requiring additional judgment algorithms, thus reducing design complexity. The image processing implementation process can be referred to the description in the corresponding section of the method embodiment below; it will not be detailed here.

[0086] In practical applications, for example Figure 3a and Figure 3b The two electronic device hardware structures shown integrate a southbridge chip (PCH) that can connect to the configuration interface of the image enhancement processing device 200, such as an I3C interface, a QSPI (Quad Serial Peripheral Interface) interface, or a UART (Universal Asynchronous Receiver / Transmitter) interface, thereby enabling the functional configuration of the image enhancement processing device 200. Therefore, the CPU can send the received configuration information to the PCH, which then transmits it to the image enhancement processing device 200 through the connected low-speed communication interface, thus configuring the image enhancement processing device 200's functions, such as image enhancement algorithm configuration and image processing method program configuration. This configuration can be determined based on image processing requirements, and will not be detailed in this embodiment.

[0087] In some other embodiments proposed in this application, such as Figure 4As shown, the image enhancement processing device 200 described above may include a data transmission port 210 and an image enhancement processor 220. Based on the descriptions of the hardware structures formed by different connection methods between the image enhancement processing device 200, the processing device 100, and the display device 300 in the above embodiments, it is clear that the type and communication principle of the data transmission port 210 often differ in different hardware structures of electronic devices.

[0088] In such Figure 3a In the hardware structure shown, the data transmission port 210 may include a unidirectional communication interface, such as an eDP interface. In this case, the display device 300 may include a display panel that supports this type of unidirectional communication interface, such as an eDP panel. Therefore, the data transmission port 210 can be connected to the display selector 130 of the display device 300 and the processing device 100 respectively, and the communication channel formed by the eDP interface connection can realize unidirectional transmission of images.

[0089] In such Figure 3b In the hardware structure shown, the data transmission port 210 may include a bidirectional communication interface, such as a PCIE (peripheral component interconnect express, high-speed serial computer expansion bus standard) interface, or a communication interface corresponding to a high-speed bidirectional communication channel configured according to application requirements, to meet the high-speed transmission requirements of various data (such as network video, local video, games, etc.). This application does not limit the configuration parameters of the high-speed bidirectional communication channel, which can be determined according to the communication requirements.

[0090] Optionally, to meet the high-speed data transmission requirements of image processing applications and reduce waiting time, the clock of the high-speed bidirectional communication channel can be configured to 16GHz and the transmission bandwidth to 8000MB / s. Based on this, the configuration parameters of the data transmission port 210 and the corresponding bidirectional communication interface of the main processor 120 of the processing device 100 can be set so that the communication requirements of the bidirectional communication channel can be met after the two are connected. However, the communication configuration parameters are not limited to those described in this embodiment and can be flexibly adjusted according to communication needs.

[0091] Therefore, it can be seen that the high-speed bidirectional communication channel adopted in this embodiment realizes the data transmission between the image enhancement processing device 200 and the processing device 100, which not only meets the data interaction requirements between the two and improves stability, but also maximizes the parallel processing capability of the graphics processor 110, greatly reduces image processing latency, and achieves real-time low-latency image processing effect.

[0092] The image enhancement processor 220 can be connected to the data transmission port 210 to receive the image to be displayed transmitted through the data transmission port 210. If the image to be displayed meets the image enhancement conditions, it performs enhancement processing on the image to be displayed to obtain the target display image; if the image to be displayed does not meet the image enhancement conditions, it directly determines the image to be displayed as the target display image. Therefore, in the different hardware structures described above, after obtaining the rendered image to be displayed, the image enhancement processor 220 can first determine whether to perform enhancement processing on the received image to be displayed, that is, selectively perform image enhancement processing operations. This ensures the quality of the output image while avoiding enhancement processing on images to be displayed that do not meet the image enhancement conditions, thus reducing the output image quality and wasting computing resources. The functional implementation process of the image enhancement processor 220 can be referred to the description in the corresponding section of the method embodiment below, which will not be detailed here.

[0093] The target display image obtained by the image enhancement processor 220 can be sent to the data transmission port 210. It should be noted that the connection and communication method between the data transmission port 210 and the image enhancement processor 220 varies depending on the hardware architecture of the electronic device. For example... Figure 3b In the hardware architecture shown, because the data transmission port 210 is a bidirectional communication interface, this bidirectional communication interface connects to the main processor 120 of the processing device 100 to form a high-speed bidirectional communication channel. This allows the image to be displayed after rendering by the processing device 100 to be transmitted to the bidirectional communication interface of the image enhancement processing device 200. The bidirectional communication interface then sends the received image to be displayed to the image enhancement processor 220. The target display image obtained by the image enhancement processor 220 can be sent to this bidirectional communication interface, transmitted to the main processor 120 via the high-speed bidirectional communication channel, and then forwarded to the display device 300 for display via the display selector 130.

[0094] In such Figure 3a In the hardware structure shown, the data transmission port 210 is a one-way communication interface. In order to realize the data interaction requirements between the processing device 100 and the display device 300, the data transmission port 210 can usually include multiple one-way communication interfaces. According to the data transmission direction, they can be divided into uplink interfaces and downlink interfaces, such as eDP RX interface, eDPTx interface, etc. The eDP RX interface is connected to the display selector 130 of the processing device 100 and receives the image to be displayed after being rendered by the processing device 100. The eDP RX interface is connected to the image enhancement processor 220 for processing. After obtaining the target display image, it can be sent to the eDP RX interface. Through the connected one-way communication channel, the target display image is transmitted to the connected display device 300 for display.

[0095] Therefore, it can be seen that in such Figure 3b In the illustrated electronic device hardware structure, the image enhancement processing device 200 can feed back the obtained target display image to the processing device 100 through the data transmission port 210, and the processing device 100 will then send the target display image to the connected display device 300; in such cases... Figure 3a In the hardware structure shown, the data transmission port 210 of the image enhancement processing device 200 is connected to the display device 300, and the image enhancement processing device 200 will directly send the obtained target display image to the display device 300 for display.

[0096] In some embodiments, such as Figure 5a As shown, the image enhancement processor 220 may include an image preprocessing unit 221 and an image enhancement processing unit 222. In this embodiment, the image preprocessing unit 221 can be connected to a data transmission port 210, receive the image to be displayed transmitted by the data transmission port 210, determine whether the image to be displayed meets the image enhancement conditions, determine the image to be displayed that does not meet the image enhancement conditions as the target display image, and send the target display image to the data transmission port 210. Under different hardware structures of electronic devices, the connection method between the data transmission port 210 and the image preprocessing unit 221 is different. Refer to the description in the corresponding part of the context; this embodiment will not elaborate further.

[0097] In this embodiment, the image preprocessing unit 221 can be a pre-processing IP chip, which can intelligently detect user behavior based on artificial intelligence (AI) technology. It detects whether the received image to be displayed meets the image enhancement conditions. If not, it can enter bypass mode, eliminating the need for enhancement processing on that frame of the image to be displayed, and directly using it as the target display image, thus reducing power consumption and improving response speed. If the image enhancement conditions are met, it can also intelligently detect enhancement regions in the image to be displayed based on AI technology, achieving local image enhancement processing and reducing computational workload.

[0098] Optionally, during the preprocessing of the received image to be displayed, the image preprocessing unit 221 may use optical flow method. This image processing optical flow method improves image processing efficiency and processing effect and expands the image processing scenario compared with pixel point processing or encoding and decoding processing.

[0099] In some other embodiments, the detection process for whether the image to be displayed meets the image enhancement conditions can also be executed by the main processor 120 in the processing device 100. Thus, in situations such as... Figure 3bIn the hardware structure of the electronic device shown, images to be displayed that do not meet the enhancement processing conditions can be directly identified as target display images and sent to the display device 300 for display, without needing to be sent to the image enhancement processing device 200 for further processing; in cases such as Figure 3a In the hardware structure of the electronic device shown, the main processor 120 can send the detection result of whether the image enhancement conditions are met to the image enhancement processing device 200. At this time, the image preprocessing unit 221 determines that the detection result is not met and directly sends the received image to be displayed as the target display image to the display device 300. This reduces the workload of the image preprocessing unit 221, saves the computing resources of the image enhancement processing device 200, and improves the image processing efficiency.

[0100] The image enhancement processing unit 222 can be connected to the image preprocessing unit 221 and the data transmission port 210 to enhance the image to be displayed that meets the image enhancement conditions, obtain the target display image, and send the target display image to the data transmission port 210 so that, under different hardware architectures, the data transmission port 210 can feed back to the processing device 100 or send it directly to the display device 300.

[0101] For example, in conjunction with the above description of data transmission port 210, in such... Figure 5b In the hardware architecture shown, the data transmission port 210 in the image enhancement processing device 200 may include a first data transmission port 211 with bidirectional data transmission function, such as the aforementioned PCIE interface or a custom high-speed bidirectional channel interface, etc. In this case, the first data transmission port 211 can be connected to the main processor 120 of the processing device 100, and the display selector 130 of the processing device 100 is connected to the display device 300. During image processing, the first data transmission port 211 receives the image to be displayed sent by the main processor 120, sends the image to be displayed to the image preprocessing unit 221, and if it is determined that the image to be displayed meets the image enhancement conditions, it is sent to the image enhancement processing unit 222 for enhancement processing. After obtaining the target display image, the received target display image is fed back to the main processor 120 of the processing device 100 through the first data transmission port 211, and then transmitted to the display device 300 through the display selector 130 of the processing device 100.

[0102] In such Figure 5cIn the hardware structure shown, the data transmission port 210 of the image enhancement processing device 200 may include a second data transmission port 212 with unidirectional data transmission function, such as the aforementioned eDP or other unidirectional communication interface. In order to meet the data transmission requirements of the image enhancement processing device to different devices of the electronic device, the second data transmission port 212 may include an uplink communication interface and a downlink communication interface, which are used to connect the display selector 130 of the processing device 100 and the display device 300 respectively, so that the uplink communication interface of the second data transmission port 212 can receive the image to be displayed output by the processing device 100. After being processed by the image enhancement processor 220, the obtained target display image is sent to the downlink communication interface, which then sends the received target display image to the connected display device 300.

[0103] In yet other embodiments, reference is made to Figure 6a and Figure 6b The schematic diagram of another optional example of the electronic device shown, in conjunction with the above description of the image enhancement processor 220, indicates that the image enhancement processing unit 222 may include at least one image enhancement algorithm program module for image enhancement processing, an acceleration engine (such as an NN engine) for accelerating the enhancement algorithm, and, if necessary, an image post-processing unit (denoted as Post-Process IP) for verifying and correcting the enhanced image to avoid introducing interference information into the image enhancement algorithm and reducing image quality. This application does not limit the composition and function of the image enhancement processing unit 222; it can be determined as appropriate.

[0104] Optional, such as Figure 6a and Figure 6b As shown in the above description of the structure of the image enhancement processor 220 in the various embodiments, the image enhancement processing unit 222 contains image enhancement algorithms that can be implemented based on deep learning, such as image enhancement algorithms based on MEMC (Motion Estimation and Motion Compensation), HDR (High Dynamic Range) algorithms, SR (Super-Resolution) algorithms, etc. This application does not limit the type of image enhancement algorithm contained in the image enhancement processing unit 222, and can be flexibly configured and adjusted according to the actual situation.

[0105] It is understood that the image enhancement chip in the image enhancement processing device 200 often includes a core unit that supports its operation, i.e., a core. This application does not elaborate on the number of cores and their functions in the image enhancement chip.

[0106] In some other embodiments proposed in this application, reference is made to Figure 6a and Figure 6b As shown, the electronic device proposed in this application may further include multiple memories 400, which can be respectively connected to the image enhancement processor 220 and the main processor 120 of the processing device 100, etc., for storing programs that implement the image processing method proposed in this application. In practical applications, the memories 400 can also be used to store various data generated during the execution of the image processing method, and can also record the enhancement processing log of the image enhancement processing device 200 executing the image processing method, the image to be displayed generated by the image rendering operation, and related rendering logs, etc. This application does not limit the type of memory 400 and the content of the stored data, and can determine it according to the application requirements.

[0107] Optional, such as Figure 6a and Figure 6b As shown, the aforementioned memory 400 may include LPDDR (Low Power Double Data Rate SDRAM) as the "working memory" of the electronic device to meet data caching requirements. In this embodiment, to meet the image data processing requirements of the image enhancement processor 220, this application may select to configure it with LPDDR4x type memory to further reduce the power consumption of the memory and its controller, improve data processing speed, and better meet image processing requirements. This application does not describe the working principle of LPDDR4x memory in detail. Of course, the memory used to store the information or data generated during the image processing method executed by the image enhancement processing device 200 is not limited to LPDDR4x type memory. Other DDR (Double Data Rate) type memory such as LPDDR4 or other types of memory different from DDR type may also be used, depending on the circumstances.

[0108] Optionally, during image processing, the image enhancement processing device 200 can construct a buffer queue to perform time-interleaved processing on each received frame of images to be displayed, without waiting for one frame to be processed and fed back to the CPU before processing the next frame. Since this application can also process partial images without waiting for the entire image to be rendered, the frames of images to be displayed in the queue may need to be processed before being combined into a complete target display image and fed back to the CPU for output. The storage of these images to be processed or processed can also be achieved using Flash memory (Flash EEPROM Memory). This application can use QSPI Flash, i.e., Flash memory supporting the QSPI protocol, which supports a queue transfer mechanism. The data processing principle of QSPI Flash is not detailed in this application.

[0109] It should be understood that the processors such as GPUs and CPUs included in the processing device 100 of the electronic device can also be configured with corresponding memory to meet the data storage requirements of the corresponding processors. As shown in Figure 5, the GPU graphics processor can be configured with GDDR (Graphics Double Data Rate) memory, and the main processor CPU can be configured with DDR memory, but it is not limited to the memory type described in this embodiment.

[0110] In conjunction with the electronic device hardware structure described in the above embodiments, in order to support the respective processing requirements of the image enhancement processing device 200 and the processing device 100 and to realize the image processing method proposed in this application, the electronic device can be configured with a corresponding driving device 500, thereby realizing the functional configuration of the structural composition of the image enhancement processing device 200 and the functional configuration of the processing device 100 in cooperation with the image processing method.

[0111] Based on this, refer to Figure 7 The schematic diagram shown illustrates the electronic device driver architecture applicable to the image processing method proposed in this application. The driver architecture of the electronic device may include: an image enhancement driver layer configured in the image enhancement processing device 200, and a system driver layer configured in the main processor 120 of the processing device 100. These two driver layers can be configured to each other to implement the functional configuration of the image enhancement processing device 200 and realize the image processing method proposed in this application. This application does not limit the driver assembly implementation process. It should be noted that, in addition to the system driver layer configured to implement the image processing method of this application, the main processor 120 may also include existing driver layers for implementing other application controls of the electronic device; these will not be detailed here.

[0112] like Figure 7As shown, in the driving device of the electronic device, a client interface and a server interface for implementing the image processing method proposed in this application can be configured, namely, an AI (Artificial Intelligence) Client API (Application Programming Interface) and a corresponding AI Server API. The AI ​​Server API can be deployed in the image enhancement driving layer, and the AI ​​Client API can be deployed in the system driving layer of the processing device 100, so that the image enhancement processing device 200 and the processing device 100 can communicate through the AI ​​Server API and the AI ​​Client API to realize the interaction of signals, data, instructions and other information between the two. The implementation process can be determined according to the image processing requirements, which will not be described in detail in this application.

[0113] Combination Figure 3b , Figure 5b as well as Figure 6a In the image processing process proposed in this application, during the image rendering operation, the processing device 100 can execute the hook D3D phPresentCb function (i.e., a display function used to send the obtained image to be displayed to the display device for display). When the GPU finishes rendering the image and outputs the image to be displayed, the AI ​​Client API can intercept the image to be displayed that the function will output and send it to the AI ​​Server API in the image enhancement driver layer of the image enhancement processing device 200. The AI ​​Server API writes the image to be processed by the image preprocessing unit and the various image enhancement algorithms contained in the image enhancement processing unit to obtain the target display image. The AI ​​Server API then feeds back the target display image to the D3D phPresentCb function of the AI ​​Client API for execution, and sends the target display image to the display device 300 for output.

[0114] As can be seen, this application does not require GPU / CPU vendor driver cooperation in the processing device 100, thus maximizing the exclusivity of the developer and vendor of the aforementioned image enhancement processing device 200. Regarding the interaction process between the two APIs, the interaction signaling steps for implementing the image processing method proposed in this application can be referred to the description in the corresponding section of the method embodiment below, which will not be detailed here.

[0115] In the aforementioned image enhancement driving layer, such as Figure 7As shown, firmware can also be configured to control the implementation process of the image processing method, as well as the configuration process and signal processing of each structural component included in the image enhancement processing device 100. This application does not limit the configuration implementation method of the firmware. The image enhancement driver layer can also be configured with a boot program module Bootrom to implement functions such as power-on timing control of each structural component module included in the image enhancement processing device 100 during the power-on process of the electronic device. The implementation method of its function will not be described in detail in the embodiments of this application.

[0116] In the aforementioned system driver layer, such as Figure 7 As shown, the system driver may include a UMD (User Mode Driver) and a KMD (Kernel Mode Driver). The aforementioned AIClient API can be deployed in the UMD to establish a communication connection with the AIServer API configured in the image enhancement driver layer of the image enhancement processing device 200, thus satisfying the data interaction between the two devices. Furthermore, the UMD may also include a chip management module and an AI adaptation layer.

[0117] The aforementioned chip management module can be used to manage or control the configuration of each structural component of the image enhancement processing device 200, such as managing the kernel configuration and the configuration of at least one required image enhancement algorithm, to meet image processing configuration requirements. The AI ​​adaptation layer can be used to process attributes such as the size and format of the image to be transmitted. For example, by scaling down the rendered image, the resources consumed during image transmission can be reduced, the transmission speed of the image to be displayed can be increased, and the computational load of the image enhancement processing can be reduced.

[0118] The KMD can be used to configure drivers for communication interfaces, such as PCIE Chip EP Driver and I3C / QSPI / UART / GPIO (General Purpose Input / Output Port) drivers. These two types of drivers may be used for image transmission and reception processing, configuration of signal timing / interrupts for different frame image processing, and determination of signal integrity. This application does not describe in detail the working principles of various interface drivers.

[0119] Based on the above description of the communication methods between the image enhancement processing device and the processing device, such as high-speed bidirectional communication or unidirectional communication, and the description of the low-speed communication interface used to implement the configuration of the image enhancement processing device, the image enhancement processing device 200 and the processing device 100 are respectively configured with corresponding types of communication interfaces, such as... Figure 7 As shown, drivers for the EP series of PCIe interfaces, and drivers for low-speed communication interfaces such as I3C / QSPI / UART are provided to support the data transmission and reception requirements of the corresponding communication interfaces.

[0120] The implementation process of the image processing method proposed in this application will be described below in conjunction with the electronic device hardware structure and driver architecture described above. However, it is not limited to the image processing method steps described in the method embodiments below. The content and order of the execution method steps can be flexibly adjusted according to actual needs. This application will not provide detailed examples of each step.

[0121] Reference Figure 8 This is a schematic flowchart of an optional example of the image processing method proposed in this application. This method can be executed by an electronic device. The composition and structure of the electronic device can be referred to the hardware structure described in the above electronic device embodiment. This embodiment describes the image processing method from the perspective of the image enhancement processing device 200. For example... Figure 8 As shown, the method may include, but is not limited to:

[0122] Step S81: Obtain the image to be displayed;

[0123] In this embodiment, the image to be displayed can be an image rendered by a processing device in an electronic device. Depending on the hardware structure of the electronic device, it can be an image extracted from the main processor of the processing device that will be output to the display device, or an image output by the display selector of the processing device, etc. This application does not limit the method of obtaining step S81 and can be determined as appropriate.

[0124] It should be noted that the image to be displayed can be a complete frame image after image rendering, or a frame image containing part of the content after image rendering. That is, this application does not limit the rendering of a complete frame image before output. It can determine the image containing part of the content to be rendered based on the content of the rendered image and output it as the image to be displayed. This application does not limit the image content of the image to be displayed obtained by the image enhancement processing device.

[0125] Step S82: Determine that the image to be displayed meets the image enhancement conditions, perform enhancement processing on the image to be displayed, and obtain the target display image;

[0126] Step S83: Determine that the image to be displayed does not meet the image enhancement conditions, and determine the image to be displayed as the target display image;

[0127] Based on the above description of the technical solution of this application, the rendered image does not necessarily need to be enhanced. Therefore, in order to reduce the computational load of image processing and ensure the image display quality, image enhancement conditions can be pre-configured to characterize the image type that needs to be enhanced. The image type can be determined based on the image content, application scenario type, source application type, image display environment type, etc. This application does not limit the content of the image enhancement conditions and the configuration implementation method, and can be determined as appropriate.

[0128] Based on this, after rendering the image to be displayed, it can be detected whether the image to be displayed meets the image enhancement conditions. If it does, the image enhancement processor of the image enhancement processing device can continue to enhance the image to be displayed; if it does not meet the conditions, the image enhancement processor can avoid spending time and resources to enhance the image to be displayed, thus avoiding the introduction of additional interference information and reducing the image display quality.

[0129] It should be noted that the detection process for whether the image to be displayed meets the image enhancement conditions can be implemented by the main processor CPU of the processing device, so as to utilize the CPU's computing power, improve processing efficiency, and save the resources of the image enhancement processor. Of course, the detection process can also be implemented by the image enhancement processor, such as by the image preprocessing unit using image recognition algorithms and machine learning algorithms included in artificial intelligence to detect each received image to be displayed and determine whether it meets the image enhancement conditions. The implementation process will not be described in detail in this embodiment of the application.

[0130] Step S84: Output the target display image.

[0131] Based on the description of the above electronic device embodiments, since the communication connection methods between the image enhancement processing device and the processing device are different under different hardware structures of the electronic device, the image transmission channels formed are different. After the image enhancement processing device obtains the target display image according to the above method, it can send the target display image to the display device for display according to the image transmission channel formed by the hardware structure of this electronic device.

[0132] In this context, for the image enhancement processing device, based on the various electronic device hardware structures described above, it can output the target display image to the main processor CPU through a bidirectional transmission interface (such as PCIE or other predefined high-speed bidirectional communication interfaces), and then the CPU executes the display function to send the target display image to the display device. When the image to be displayed is received by a unidirectional communication interface (such as eDP), the obtained target display image can be sent to the connected display device through a downlink unidirectional communication interface. The transmission link of the target display image can be determined by referring to the above hardware structure diagram, and will not be described in detail in this embodiment.

[0133] Reference Figure 9 This is a flowchart illustrating another optional example of the image processing method proposed in this application. This method can still be described from the perspective of an image enhancement processing device in an electronic device. It can be an optional refinement of the image processing method described above, but is not limited to the refinement method described in this embodiment. Referring to the above... Figure 3b , Figure 5b as well as Figure 6a The hardware structure of the electronic device shown, and Figure 7 The schematic diagram of the electronic device driving architecture shown in this embodiment illustrates that the image processing method proposed in this embodiment may include:

[0134] Step S91: Obtain the image to be displayed after being rendered by the processing device;

[0135] In this embodiment, the processing device performs an image rendering operation to obtain an image to be displayed. When executing the display function (i.e., sending the image to be displayed to the display device for display), it intercepts the function using the AI ​​Client API (client interface) in the system driver layer deployed by the CPU (which belongs to the driver of the image enhancement processing device), extracts the image to be displayed to be sent to the display device, i.e., hooks the D3D phPresentCb function, and sends the intercepted image to be displayed to the AI ​​Server API (service interface) in the image enhancement driver layer of the image enhancement processing device, so that the image enhancement processing device can obtain the image to be displayed.

[0136] For any image frame to be displayed received by the AI ​​Server API of the image enhancement processing device, such as Figure 10 As shown, data can be written to the pending queue configured in the firmware for caching, such as... Figure 10 The Frame1, Frame2, etc. in the image enhancement processor will sequentially extract one frame of the image to be displayed and perform detection processing to determine whether the image enhancement conditions are met.

[0137] Step S92: Obtain the scene type to which the image to be displayed belongs;

[0138] In this embodiment, the image enhancement conditions are determined based on the scene type of the image application. For example, at least one scene that needs image enhancement processing is determined as a first type of scene, such as various video playback scenes, game operation scenes, etc. In this way, after the image enhancement processor extracts a frame of image to be displayed from the processing queue, it can determine the scene type of the image to be displayed based on the content source of the image to be displayed or by directly performing content analysis on the image to be displayed. This application does not limit the method of obtaining step S92 and can be determined as appropriate.

[0139] In some embodiments, under certain specific business scenarios, image enhancement processing can also be performed on images from a specified signal source or containing specific content, forming image enhancement conditions for that specific business scenario. In this way, after retrieving a cached frame of image to be displayed from the processing queue, the image enhancement conditions can be used to directly detect whether the image to be displayed originates from the specified signal source or whether the image content is the specified content. If so, the frame of image to be displayed satisfies the image enhancement conditions; otherwise, it does not. It is evident that the detection method for the image to be displayed differs for image enhancement conditions of different content. This application will not provide detailed examples of each method here; the method can be determined based on the scenario requirements.

[0140] Step S93: If the scene type belongs to the first type of scene, identify the image region to be enhanced contained in the image to be displayed;

[0141] Step S94: Enhance the image region to be enhanced contained in the image to be displayed to obtain the target display image;

[0142] Once the image enhancement processor determines that an extracted frame of the image to be displayed meets the image enhancement conditions, it can perform enhancement processing on it. Optionally, to reduce the computational load of image enhancement processing and ensure the image enhancement effect, this application proposes to enhance the regions in the image to be displayed that require enhancement processing, while not enhancing the regions that do not require enhancement processing. This avoids performing enhancement operations on regions that do not require enhancement processing, introducing additional interference information, and reducing the quality of the enhanced image. This application proposes to detect the image regions in the image to be displayed that require enhancement processing, which may be at least a portion of the image region to be displayed. Then, a suitable deep learning-based image enhancement algorithm is selected to enhance the image regions to be enhanced, thereby obtaining the target display image.

[0143] For example, refer to Figure 11aThe image shown is a frame to be displayed. When the user watches the video, the video playback interface is displayed in a thumbnail state, so that the video playback interface is displayed in part of the display area of ​​the electronic device's display device, while the application desktop image is displayed in another part of the display area. In this case, for the display as shown... Figure 11a The image shown only requires enhancement processing on the video image area of ​​the video playback interface; the application desktop image does not need to be enhanced, thus improving algorithm efficiency and reducing power consumption and processing time.

[0144] Based on this, for example Figure 11a Enhanced region detection can be performed on the image to be displayed, as shown below, to obtain... Figure 11b The detection results shown can determine the image region to be enhanced in the image to be displayed based on the pixel values ​​of each pixel. This application does not limit the implementation method of detecting the image region to be enhanced, and is not limited to it. Figure 11b The detection and processing results are shown. Afterwards, according to the detection results, this application can segment a frame of the image to be displayed into several image regions, perform enhancement processing on the image regions that require enhancement processing, and directly output the regions that do not require enhancement processing. Thus, in situations such as... Figure 10 In this process, Frame1 is divided into three parts and processed to obtain Frame1_1, Frame1_2, and Frame1_2. These three parts can then be combined to form a complete target display image output, but this method of image processing is not limited to it.

[0145] In some embodiments, based on the image enhancement processing method proposed in this application, an image enhancement model can be pre-trained using a deep learning image enhancement algorithm to identify the region of the input image to be enhanced and perform enhancement processing on that region. Thus, after obtaining a frame of image to be displayed, the image enhancement processor can directly input the image to be displayed into the matching image enhancement model and output the enhanced target display image. The training implementation method of the image enhancement model is not detailed in this application; it can be determined based on the operational principles of the selected deep learning-based image enhancement algorithm.

[0146] It should be understood that if the user controls the video player of an electronic device in full-screen mode, the video image will be displayed full-screen on the electronic device's screen. For images to be displayed in this output mode, the entire image area is the area to be enhanced, and all image content of the image to be displayed needs to be enhanced. The method for detecting the image area to be enhanced in this type of image can still be determined according to the method described above. Alternatively, the output mode and application source of the image to be displayed can be detected first. If the application source is included in the first scenario type of full-screen output, the image to be displayed can be directly enhanced without having to identify the image area to be enhanced using the method described above, thus saving computational resources.

[0147] Furthermore, in practical applications, the enhancement processing methods for different frames of the image to be displayed can be different. That is, the deep learning-based image enhancement algorithms used can be different or the same, depending on the scene to which the image to be displayed belongs. It should be noted that if the image region to be enhanced in a frame is a part of the image region to be displayed, after enhancing that part of the image region, the enhanced part of the image region and the unenhanced part of the image region constitute the target display image.

[0148] Optionally, for the enhanced image to be displayed, this application can also perform calibration processing by the image post-processing unit, determine the calibrated image as the target display image, and write it into the output queue for caching. This application does not limit the implementation method of image calibration after enhancement processing, and can be determined as appropriate.

[0149] Step S95: If the scene type belongs to the second type of scene, the acquired image to be displayed is determined as the target display image;

[0150] Based on the above description of image enhancement conditions, the image to be displayed is determined. If a frame of image to be displayed extracted from the queue does not meet the image enhancement processing conditions, in order to avoid introducing additional information and affecting image quality, it is not necessary to enhance the image to be displayed, and the image to be displayed is directly determined as the target display image.

[0151] In some embodiments of this application, since the image enhancement processing device is equipped with a buffer queue to cache the acquired image to be displayed, and after processing the image to be displayed, it can be written into another buffer queue for storage, this application can adjust the image processing timing of different chips in the electronic device, so that the timing of different processing processes of adjacent frames overlaps, that is, there is an overlap in the processing time of the images to be displayed in adjacent frames, and the processing content of the images to be displayed in adjacent frames is different during the overlap processing time. Compared with the timing processing method of processing one frame of images to be displayed and outputting it to the display device before processing the next frame of images to be displayed, this greatly improves the image processing efficiency, realizes full utilization of the computing resources of the electronic device, and reduces the time loss in the image transmission process.

[0152] For example, refer to Figure 12The illustrated timing flow diagram of the image processing method shows that for data such as game data and video streams, the input is processed by the graphics processing unit (GPU) for image rendering. An image rendering model can be used to draw / render the input data, and the rendered image is sent to the main processor (CPU) via PCIe. In the hardware architecture of this embodiment, the CPU can send the received image to be displayed to the AI ​​Server API of the image enhancement processing device (i.e., an AI chip) via the AI ​​Client API, where it is written into a cache queue (such as the processing queue mentioned above) for storage. Afterwards, the image preprocessing unit can perform preprocessing frame by frame, and the acceleration engine calls the required image enhancement algorithm for enhancement processing. The resulting target display image is written into another cache queue for storage. Then, the cached target display image can be read frame by frame, and the obtained target display image is sent to the CPU's AI Client API via the AI ​​Server API. The CPU then sends the target display image to the display device via eDP.

[0153] In the above processing, such as Figure 12 As shown, the processing steps of different frames of images to be displayed on different chips (such as GPU, CPU, AI chip, etc.) can overlap. In adjacent processing steps, on different chips, such as when the CPU is transmitting the nth frame of the image to be displayed, the GPU can render the (n+1)th frame of the image. During this period, preprocessing of the (n-1)th frame of the image to be displayed may be performed. Therefore, when designing the processing sequence of consecutive frames of images based on information such as the processing entity and processing time of each frame, the rendered image can be transmitted to the image enhancement processing device for processing while the GPU is rendering a portion of the image, without waiting for all images to be rendered, greatly reducing the time loss during image transmission.

[0154] It should be noted that the design schemes for processing different frames of images in different chips include, but are not limited to, the following: Figure 12 The timing sequence shown can be flexibly adjusted according to the actual situation, such as the overlap duration, etc. This application will not provide detailed examples here.

[0155] Step S96: The target display image is fed back to the processing device, which then sends the target display image to the display device for display.

[0156] Referring to the applicable electronic device hardware structure in this embodiment, after processing the intercepted image to be displayed, the image enhancement processing device can feed back the obtained target display image to the CPU of the processing device, so that the CPU executes the display function and sends the target display image to the display device for display. The driving process for image processing and transmission can be referred to the description in the corresponding part of the above embodiment, and will not be repeated here.

[0157] In summary, in this embodiment of the application, the electronic device is configured with an image enhancement processing device as described above, and is connected to the main processor of the processing device via high-speed bidirectional communication. When the processing device performs image rendering and outputs the rendered image to be displayed, the output operation of the display function on the image to be displayed is intercepted. Instead, the image to be displayed is sent to the image enhancement processing device through the high-speed bidirectional communication channel. The image area to be enhanced is enhanced according to the image enhancement conditions. The high-quality target display image after enhancement is fed back to the main processor through the high-speed bidirectional communication channel. The main processor then sends the target display image to the display device for display, thus ensuring the quality of the displayed image.

[0158] Furthermore, this application employs a high-speed bidirectional communication channel to achieve data interaction between the image enhancement processing device and the processing device of the electronic device system, thereby improving image processing stability and maximizing the parallel processing capabilities of the graphics processing unit (GPU). Figure 12 The timing flow shown greatly reduces algorithm latency, achieving real-time and low-latency image processing.

[0159] For images to be displayed that do not require enhancement processing, such as desktop images and text document interface images, they can directly enter bypass mode. In this case, the image enhancement processing unit can stop working and the image to be displayed after GPU rendering can be sent to the display device for output. This solves the technical problem that performing indiscriminate enhancement processing on all rendered images leads to the introduction of additional interference data and reduces the quality of the output image.

[0160] In some other embodiments proposed in this application, such as Figure 3a , Figure 5c as well as Figure 6b In the illustrated electronic device hardware structure, the image enhancement processing device can receive the image to be displayed output by the processing device after rendering. Data transmission between the image enhancement processing device and the processing device can be achieved through the eDP communication interface. Compared to USB or MIPI communication methods, this application is applicable not only to processing images captured by a camera but also to enhancing network / local video and game images. The processing procedure of the image enhancement processing device on the received image to be displayed can be referred to, but is not limited to, the description in the corresponding parts of the above embodiments; this embodiment will not elaborate further here.

[0161] In summary, based on the above analysis, the image enhancement processing device can detect the image processing area that needs enhancement when it receives any frame of the image to be displayed, perform enhancement processing on it, and send the resulting target display image to the display device for display. This allows for enhancement processing of low-quality image content (such as adjusting the image color temperature and color to improve the image color presentation effect), while high-quality image content does not require enhancement processing. This greatly avoids introducing additional image problems during the enhancement process. Furthermore, the enhancement algorithm is only executed on the image processing area to be enhanced, which greatly improves the efficiency of the enhancement algorithm, reduces power consumption and processing time, and enhances the user experience.

[0162] For the image processing methods described in the above embodiments, in order to identify the image region to be enhanced within the image to be displayed, an attention mechanism can be used to determine the region of interest (ROI) within the image to be displayed as the image region to be enhanced. This ROI may include the first application content in the first type of scenario, such as video images, game images, etc. According to this technical configuration, this application can train an image enhancement recognition model based on an attention mechanism. Thus, the image to be displayed can be directly input into the image enhancement recognition model, and the image region to be enhanced can be output. This application does not limit the training method of this image enhancement recognition model.

[0163] In some other embodiments, based on the above analysis, this application can directly analyze the content of the image to be displayed, determine that the content of the image to be displayed belongs to the first application content, and determine the entire display area of ​​the image to be displayed as the image area to be enhanced, such as identifying the entire display area of ​​a video image / game screen as the image area to be enhanced. In some other application scenarios, if this application determines that the image to be displayed contains both the first application content and the second application content, such as... Figure 11a The image to be displayed can identify the area where the first application content is located as the image area to be enhanced, such as identifying the display area where the video image is located as the image area to be enhanced. This application does not limit the implementation method of image content recognition. It can be determined by an image recognition algorithm based on the attention mechanism, or by the pre-constructed correspondence between different application contents and application display areas, to determine the application display area of ​​the first application content as the image area to be enhanced, etc.

[0164] Reference Figure 13 The diagram below shows an optional example of the image processing apparatus proposed in this application, which may include:

[0165] Image acquisition module 131 is used to acquire the image to be displayed;

[0166] The first determining module 132 is used to determine that the image to be displayed meets the image enhancement conditions, and to perform enhancement processing on the image to be displayed to obtain the target display image;

[0167] The second determining module 133 is used to determine that the image to be displayed does not meet the image enhancement conditions, and to determine the image to be displayed as the target display image;

[0168] The target display image output module 134 is used to output the target display image.

[0169] In some embodiments, such as Figure 3a Under the hardware structure of the electronic device shown, the above-mentioned image acquisition module may include:

[0170] The image receiving unit is used to receive the image to be displayed after being rendered by the processing device.

[0171] Accordingly, the aforementioned target display image output module may include:

[0172] An image sending unit is used to send the target display image to a display device for display.

[0173] In some other embodiments, such as Figure 3b Under the hardware structure of the electronic device shown, the above-mentioned image acquisition module may include:

[0174] The image acquisition unit is used to acquire the image to be displayed after being rendered by the processing device, that is, the rendered image to be sent to the display device.

[0175] An image feedback unit is used to feed back the target display image to the processing device, and the processing device sends the target display image to the display device for display.

[0176] Optionally, in conjunction with the above-described device structure, the first determining module may include:

[0177] A scene type obtaining unit is used to obtain the scene type to which the image to be displayed belongs;

[0178] The first determining unit is configured to determine that the image to be displayed satisfies the image enhancement conditions if the scene type belongs to the first type of scene.

[0179] An image region recognition unit is used to identify the image region to be enhanced contained in the image to be displayed;

[0180] An enhancement processing unit is used to perform enhancement processing on the image region to be enhanced contained in the image to be displayed, so as to obtain a target display image.

[0181] The second determining module mentioned above may include:

[0182] The second determining unit is used to determine that if the scene type belongs to the second type of scene, the image to be displayed does not meet the image enhancement conditions, and to determine the image to be displayed as the target display image.

[0183] In one possible implementation, the aforementioned image region recognition unit to be enhanced may include:

[0184] The first recognition unit is configured to determine, based on an attention mechanism, that the region of interest contained in the image to be displayed is the region of image to be enhanced; the region of interest includes the first application content in the first type of scenario; or,

[0185] The second identification unit is configured to determine that the content of the image to be displayed belongs to the first application content, and to define the entire display area of ​​the image to be displayed as the image area to be enhanced; and / or,

[0186] The third recognition unit is used to determine that the image to be displayed includes first application content and second application content, and to recognize the area where the first application content is located as the area of ​​the image to be enhanced.

[0187] In some other embodiments, the image processing apparatus described above may further include:

[0188] The first caching module is used to write the obtained image to be displayed into a processing queue to determine frame by frame whether the cached image to be displayed meets the image enhancement conditions; and / or,

[0189] The second caching module is used to write the obtained target display image into the output queue, so as to output the cached target display image frame by frame; and / or,

[0190] The calibration processing module is used to calibrate the enhanced image to be displayed in order to obtain the target display image.

[0191] It should be noted that the various modules and units in the above-mentioned device embodiments can all be stored in the memory as program modules. The processor executes the above-mentioned program modules stored in the memory to realize the corresponding functions. The functions realized by each program module and its combination, as well as the technical effects achieved, can be referred to the description of the corresponding part of the above-mentioned method embodiments. This embodiment will not repeat them here.

[0192] This application also provides a computer-readable storage medium on which a computer program can be stored, which can be invoked and loaded by a processor to implement the various steps of the image processing method described in the above embodiments.

[0193] Finally, it should be noted that, regarding the above embodiments, unless the context explicitly indicates an exception, the words "a," "an," "an," and / or "the" do not specifically refer to the singular and may also include the plural. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of explicitly identified steps and elements, and these steps and elements do not constitute an exclusive list; the method or apparatus may also include other steps or elements. An element defined by the phrase "comprising an..." does not exclude the presence of other identical elements in the process, method, product, or apparatus that includes the element.

[0194] In the description of the embodiments of this application, unless otherwise stated, " / " means "or", for example, A / B can mean A or B; "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the description of the embodiments of this application, "multiple" refers to two or more.

[0195] The terms used in this application, such as "first" and "second," are for descriptive purposes only, used to distinguish one operation, unit, or module from another, and do not necessarily require or imply any such actual relationship or order between these units, operations, or modules. Furthermore, they should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include one or more of that feature.

[0196] Furthermore, the various embodiments in this specification are described in a progressive or parallel manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatuses and electronic devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to in the method section.

[0197] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image processing method, the method comprising: Obtain the image to be displayed; The image to be displayed is determined to meet the image enhancement conditions, and the image to be displayed is enhanced to obtain the target display image; wherein, the output mode and application source of the image to be displayed are detected, and if the application source is included in the first scene type of full-screen output mode, it is determined that the image enhancement conditions are met; If it is determined that the image to be displayed does not meet the image enhancement conditions, the image to be displayed is determined as the target display image; Output the target display image.

2. The method according to claim 1, wherein obtaining the image to be displayed comprises: The image to be displayed is rendered and output by the receiving and processing device; The output of the target display image includes: The target display image is sent to the display device for display. or, Obtaining the image to be displayed includes: Acquire the image to be displayed after being rendered by the processing device; The output of the target display image includes: The target display image is fed back to the processing device, which then sends the target display image to the display device for display.

3. The method according to claim 1 or 2, wherein determining that the image to be displayed satisfies the image enhancement condition includes: Obtain the scene type to which the image to be displayed belongs; If the scene type belongs to the first type of scene, the image to be displayed is determined to meet the image enhancement conditions; The image to be displayed does not meet the image enhancement conditions in the following ways: If the scene type belongs to the second type of scene, it is determined that the image to be displayed does not meet the image enhancement conditions.

4. The method according to claim 1 or 2, wherein enhancing the image to be displayed to obtain the target display image comprises: Identify the image region to be enhanced contained in the image to be displayed; The image region to be enhanced contained in the image to be displayed is subjected to enhancement processing to obtain the target display image.

5. The method according to claim 4, wherein identifying the image region to be enhanced contained in the image to be displayed includes: Based on the attention mechanism, the region of interest contained in the image to be displayed is determined as the region of image to be enhanced; The region of interest includes the first application content in the first type of scenario; or, Determining that the content of the image to be displayed belongs to the first application content, the entire display area of ​​the image to be displayed is determined as the image area to be enhanced; and / or, The image to be displayed is determined to include first application content and second application content, and the area where the first application content is located is identified as the area of ​​the image to be enhanced.

6. The method according to claim 2, further comprising: The obtained image to be displayed is written into the processing queue to determine whether the cached image to be displayed meets the image enhancement conditions frame by frame. And / or, The obtained target display image is written into the output queue to output the cached target display image frame by frame; And / or, The enhanced image to be displayed is then calibrated to obtain the target display image.

7. An image processing apparatus, the apparatus comprising: The image to be displayed module is used to obtain the image to be displayed. The first determining module is used to determine that the image to be displayed meets the image enhancement conditions, and to perform enhancement processing on the image to be displayed to obtain the target display image; wherein, the output mode and application source of the image to be displayed are detected, and if the first scene type of the full-screen output mode contains the application source, it is determined that the image enhancement conditions are met. The second determining module is used to determine that the image to be displayed does not meet the image enhancement conditions, and to determine the image to be displayed as the target display image; The target display image output module is used to output the target display image.

8. An electronic device, the electronic device comprising: A processing unit for performing image rendering operations to obtain an image to be displayed; An image enhancement processing apparatus for implementing the image processing method as described in any one of claims 1-6; A display device is used to display the image enhancement processing device or the target display image output by the processing device.

9. The electronic device according to claim 8, wherein the image enhancement processing apparatus comprises: A data transmission port is used to connect to the processing device and receive the image to be displayed output by the processing device; An image enhancement processor is configured to connect to the data transmission port, receive the image to be displayed transmitted by the data transmission port, determine that the image to be displayed meets the image enhancement conditions, perform enhancement processing on the image to be displayed to obtain a target display image; determine that the image to be displayed does not meet the image enhancement conditions, determine the image to be displayed as the target display image, and send the target display image to the data transmission port. The data transmission port is also used to send the target display image to the processing device, which then sends the target display image to the connected display device; or it is also used to connect to the display device and send the target display image to the display device.

10. The electronic device of claim 9, wherein the image enhancement processor comprises: An image preprocessing unit is configured to connect to the data transmission port, receive the image to be displayed transmitted by the data transmission port, determine whether the image to be displayed meets the image enhancement conditions, determine the image to be displayed that does not meet the image enhancement conditions as the target display image, and send the target display image to the data transmission port. An image enhancement processing unit is used to connect the image preprocessing unit and the data transmission port, enhance the image to be displayed that meets the image enhancement conditions, obtain a target display image, and send the target display image to the data transmission port; The data transmission port includes a first data transmission port with bidirectional data transmission function. The first data transmission port is connected to the processing device to receive the image to be displayed output by the processing device and send the received target display image to the processing device. Alternatively, the data transmission port may include a second data transmission port with unidirectional data transmission function, which is connected to the processing device and the display device respectively, to receive the image to be displayed output by the processing device and send the received target display image to the display device.

Images