Image Processing Method, System, Electronic Device, and Computer-Readable Storage Medium

Abstract

This application is applicable to the field of artificial intelligence technology, and provides an image processing method, system, electronic device, and computer-readable storage medium. In the image processing method of this application, the electronic device uses a first graphics rendering hardware to render the native image data to obtain a first image, and then uses a second graphics rendering hardware to perform super-resolution reconstruction on the first image to obtain a target image. The first graphics rendering hardware and the second graphics rendering hardware are different graphics rendering hardwares. Compared with the method of directly rendering the native image data into a target image using a single graphics rendering hardware, the method of this application can reduce the rendering power consumption of rendering high-quality images, reduce the amount of calculation, improve the rendering efficiency, and make full use of the hardware resources of the heterogeneous graphics rendering hardwares in the electronic device, solving the problem that in the existing image solutions, when the hardware resources of the electronic device are insufficient, the electronic device can only run high-quality products with a lower image quality.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence, and particularly relates to an image processing method, system, electronic device, and computer-readable storage medium. Background Art

[0002] Currently, there are more and more products with fine pictures on the market. For example, high-definition games and videos.

[0003] When an electronic device renders the pictures of these products, due to high rendering power consumption and large amount of calculation, it has high requirements for the hardware resources of the electronic device. However, when the hardware resources of the electronic device are insufficient, the electronic device can only run these products with a lower picture quality, seriously affecting the picture effect of the products. Summary of the Invention

[0004] This application provides an image processing method, system, electronic device, and computer-readable storage medium, which solves the problem that in the existing image scheme, when the hardware resources of the electronic device are insufficient, the electronic device can only run high-definition products with a lower picture quality.

[0005] To achieve the above object, this application adopts the following technical solutions:

[0006] In a first aspect, an image processing method is provided, which is applied to a first electronic device and includes:

[0007] The first electronic device obtains native image data, where the native image data is image data generated by an application program and not yet rendered;

[0008] The first electronic device renders the native image data through a first graphics rendering hardware to obtain a first image;

[0009] The first electronic device performs super-resolution reconstruction on the first image through a second graphics rendering hardware to obtain a target image, where the first graphics rendering hardware and the second graphics rendering hardware are different graphics rendering hardwares.

[0010] It should be noted that the application program generates native image data during operation.

[0011] After the first electronic device obtains the native image data, it can render the native image data through a first graphics rendering component, so that the native image data is converted into visible pixels to obtain a first image.

[0012] After that, the first electronic device can perform super-resolution reconstruction on the first image through a second graphics rendering component to obtain a high-quality target image.

[0013] Compared with the method of directly rendering the original image data into a high-quality target image, the image processing method of the present application obtains the target image by combining preliminary rendering and super-resolution reconstruction, which can reduce the rendering power consumption and computing amount of the electronic device for rendering high-quality images.

[0014] In addition, the specific types of the first graphics rendering component and the second graphics rendering component can be one or more of a central processing unit (CPU), a graphics processing unit (GPU), and a neural-network processing unit (NPU).

[0015] Moreover, the first graphics rendering component and the second graphics rendering component are different graphics rendering components, so as to make full use of the heterogeneous graphics rendering components inside the electronic device and avoid the electronic device running high-quality products at a lower image quality due to insufficient hardware resources of a single graphics rendering hardware.

[0016] In a possible implementation manner of the first aspect, the first electronic device performs super-resolution reconstruction on the first image through the first graphics rendering hardware to obtain a target image, including:

[0017] The first electronic device obtains the identifier of the application;

[0018] The first electronic device searches for a target super-resolution model associated with the identifier;

[0019] The first electronic device performs super-resolution reconstruction on the first image through the first graphics rendering hardware and the found target super-resolution model to obtain a target image.

[0020] It should be noted that the first electronic device can perform super-resolution reconstruction on the first image through a super-resolution model.

[0021] Among them, the super-resolution model can include a specific super-resolution model and a general super-resolution model. The specific super-resolution model is only applicable to some applications, with poor applicability but high image quality optimization ability. The general super-resolution model has high applicability but limited image quality optimization ability.

[0022] Since the application range of the specific super-resolution model is limited, the electronic device can pre-establish an association relationship between the specific super-resolution model and the application to which it is applicable.

[0023] After obtaining the first image, the first electronic device can obtain the identifier of the application and search for a target super-resolution model associated with the identifier.

[0024] If the first electronic device can find a target super-resolution model associated with the above-mentioned identifier, it means that there is a specific super-resolution model (i.e., the target super-resolution model) applicable to the above application in the first electronic device. The first electronic device can perform super-resolution reconstruction on the first image through the above-mentioned first graphics rendering hardware and the above-mentioned target super-resolution model.

[0025] When the electronic device uses the target super-resolution model to perform super-resolution reconstruction on the first image, it can better improve the image quality of the target image.

[0026] In addition, the identifier of the above application can be the package name of the application, or the above application can also be a user-defined identifier.

[0027] In a possible implementation manner of the first aspect, after the first electronic device searches for a target super-resolution model associated with the identifier, it further includes:

[0028] If the target super-resolution model associated with the identifier cannot be found, the first electronic device performs super-resolution reconstruction on the first image through the first graphics rendering hardware and a preset general super-resolution model to obtain a target image.

[0029] It should be noted that if the first electronic device cannot find a target super-resolution model associated with the above identifier, it means that there is no associated specific super-resolution model for this application.

[0030] At this time, the first electronic device can perform super-resolution reconstruction on the first image through the first graphics rendering hardware and a preset general super-resolution model to obtain a target image.

[0031] In a possible implementation manner of the first aspect, after the first electronic device performs super-resolution reconstruction on the first image through a preset general super-resolution model to obtain a target image, it further includes:

[0032] The first electronic device establishes an association relationship between the identifier and the general super-resolution model.

[0033] It should be noted that when the first electronic device uses the general super-resolution model to perform super-resolution reconstruction on the first image, since the first electronic device may be provided with multiple general super-resolution models, in order to enable the first electronic device to call the same general super-resolution model to perform super-resolution reconstruction on the first image when the application is started next time, the first electronic device can establish the above-mentioned association relationship between the identifier and the general super-resolution model.

[0034] When the application is launched next time, the first electronic device can find the general super-resolution model according to the association relationship between the above identifier and the general super-resolution model, determine the general super-resolution model as the target super-resolution model, and use the target super-resolution model to perform super-resolution reconstruction on the first image, so that the first electronic device can maintain the same image quality optimization level when processing the images of the application.

[0035] In a possible implementation manner of the first aspect, the first electronic device renders the native image data through the first graphics rendering hardware to obtain a first image, including:

[0036] The first electronic device renders the native image data through the first graphics rendering hardware and a preset first image resolution to obtain a first image.

[0037] It should be noted that when the first electronic device performs preliminary rendering, it can render the native image data according to the first image resolution. The first image resolution is preset by the first electronic device.

[0038] In a possible implementation manner of the first aspect, the first electronic device performs super-resolution reconstruction on the first image through the second graphics rendering hardware to obtain a target image, including:

[0039] The first electronic device performs super-resolution reconstruction on the first image through the second graphics rendering hardware and a single-fold enhanced super-resolution model to obtain a target image, where the image resolution of the first image is the same as the image resolution of the target image, and the single-fold enhanced super-resolution model is a super-resolution model in which the image resolution of the input image is the same as the image resolution of the output image.

[0040] It should be noted that if the image resolution of the first image is the same as the image resolution of the target image, then when the first electronic device selects a super-resolution model to perform super-resolution reconstruction on the first image, it can select a single-fold enhanced super-resolution model.

[0041] For the single-fold enhanced super-resolution model, the image resolution of the input image of the model is the same as the image resolution of the output image of the model.

[0042] In a possible implementation manner of the first aspect, the first electronic device performs super-resolution reconstruction on the first image through the second graphics rendering hardware to obtain a target image, including:

[0043] The first electronic device performs super-resolution reconstruction on the first image through the second graphics rendering hardware and a multi-fold enhanced super-resolution model to obtain a target image, where the resolution of the first image is less than the resolution of the target image, and the multi-fold enhanced super-resolution model is a super-resolution model in which the resolution of the input image is less than the resolution of the output image.

[0044] It should be noted that if the resolution of the first image is less than the resolution of the target image, the electronic device can select a multi-fold enhanced super-resolution model to perform super-resolution reconstruction on the first image.

[0045] For the multi-fold enhanced super-resolution model, the resolution of the input image of this model is less than the resolution of the output image of this model. That is to say, the multi-fold enhanced super-resolution model can increase the resolution of the input image.

[0046] In a possible implementation manner of the first aspect, the first electronic device renders the native image data through the first graphics rendering hardware to obtain a first image, including:

[0047] The first electronic device renders the native image data through a graphics processing unit to obtain a first image;

[0048] Correspondingly, the first electronic device performs super-resolution reconstruction on the first image through the second graphics rendering hardware to obtain a target image, including:

[0049] The first electronic device performs super-resolution reconstruction on the first image through a neural network processor to obtain a target image.

[0050] It should be noted that the first electronic device processes the native image data through heterogeneous graphics rendering hardware to obtain a target image. Specifically, the first electronic device can render the native image data through a graphics processing unit to obtain a first image, and perform super-resolution reconstruction on the first image through a neural network processor.

[0051] The first electronic device selects appropriate graphics rendering hardware to perform corresponding operations, which can improve the image processing efficiency of the first electronic device.

[0052] In a second aspect, a method for image processing is provided, which is applied to a second electronic device and includes:

[0053] The second electronic device receives a first image sent by the first electronic device, where the first image is an image obtained after the first electronic device renders the native image data generated by an application program;

[0054] The second electronic device performs super-resolution reconstruction on the first image to obtain a target image.

[0055] It should be noted that when the user hopes to project the screen display of the first electronic device to the second electronic device, the first electronic device can initially render the native image data of the application locally to obtain a first image.

[0056] Then, the first electronic device sends the first image to the second electronic device.

[0057] The second electronic device receives the first image and performs super-resolution reconstruction on the first image to obtain a target image with high image quality.

[0058] In the process of obtaining the target image, the first electronic device and the second electronic device jointly perform image processing, which can make full use of the hardware resources of the graphics rendering hardware of different electronic devices, thereby reducing the load on the local hardware resources of the first electronic device when rendering high-quality images, reducing the rendering consumption of the first electronic device for rendering high-quality images, and moreover, making full use of the hardware resources of different electronic devices can better improve the image quality, thus improving the user experience.

[0059] In a possible implementation manner of the second aspect, the second electronic device performs super-resolution reconstruction on the first image to obtain a target image, including:

[0060] The second electronic device obtains the identifier of the application;

[0061] The second electronic device searches for a target super-resolution model associated with the identifier;

[0062] The second electronic device performs super-resolution reconstruction on the first image through the found target super-resolution model to obtain a target image.

[0063] It should be noted that the second electronic device can perform super-resolution reconstruction on the second image through a super-resolution model.

[0064] Among them, the super-resolution model can include a specific super-resolution model and a general super-resolution model. The specific super-resolution model is only applicable to some applications, and its applicability is poor, but it has high image quality optimization ability. The general super-resolution model has high applicability, but its image quality optimization ability is limited.

[0065] Since the application range of the specific super-resolution model is limited, therefore, the electronic device can pre-establish an association relationship between the specific super-resolution model and the application to which it is applicable.

[0066] After the second electronic device obtains the second image, it can obtain the identifier of the application and search for a target super-resolution model associated with the identifier.

[0067] If the second electronic device can find the target super-resolution model associated with the above identifier, it means that there is a specific super-resolution model (i.e., the target super-resolution model) applicable to the above application in the second electronic device. The second electronic device can perform super-resolution reconstruction on the second image through the graphics rendering hardware and the above target super-resolution model.

[0068] When the electronic device uses the target super-resolution model to perform super-resolution reconstruction on the second image, it can better improve the image quality of the target image.

[0069] In addition, the identifier of the above application can be the package name of the application, or the above application can also be a user-defined identifier.

[0070] In a possible implementation manner of the second aspect, after the second electronic device searches for the target super-resolution model associated with the identifier, it further includes:

[0071] If the target super-resolution model associated with the identifier is not found, the second electronic device performs super-resolution reconstruction on the first image through a preset general super-resolution model to obtain a target image.

[0072] It should be noted that if the second electronic device cannot find the target super-resolution model associated with the above identifier, it means that there is no associated specific super-resolution model for this application.

[0073] At this time, the second electronic device can perform super-resolution reconstruction on the second image through the graphics rendering hardware and the preset general super-resolution model to obtain a target image.

[0074] In a possible implementation manner of the second aspect, after the second electronic device performs super-resolution reconstruction on the first image through a preset general super-resolution model to obtain a target image, it further includes:

[0075] The second electronic device establishes an association relationship between the identifier and the general super-resolution model.

[0076] It should be noted that when the second electronic device uses the general super-resolution model to perform super-resolution reconstruction on the second image, since the second electronic device may be provided with multiple general super-resolution models, in order to enable the second electronic device to call the same general super-resolution model to perform super-resolution reconstruction on the second image when the application is started next time, the second electronic device can establish an association relationship between the above identifier and the general super-resolution model.

[0077] When the application is launched next time, the second electronic device can find the general super-resolution model according to the association relationship between the above identifier and the general super-resolution model, determine the general super-resolution model as the target super-resolution model, and use the target super-resolution model to perform super-resolution reconstruction on the second image, so that the second electronic device can maintain the same image quality optimization level when processing the images of the application.

[0078] In a possible implementation manner of the second aspect, the first resolution of the first image is the same as the image resolution of the target image;

[0079] The second electronic device performs super-resolution reconstruction on the first image to obtain a target image, including:

[0080] The second electronic device performs super-resolution reconstruction on the first image through a single-enhanced super-resolution model to obtain the target image, where the single-enhanced super-resolution model is a super-resolution model in which the image resolution of the input image is the same as the image resolution of the output image.

[0081] It should be noted that if the first image resolution is the same as the image resolution of the target image, the second electronic device can select a single-enhanced super-resolution model when selecting a super-resolution model to perform super-resolution reconstruction on the first image.

[0082] The image resolution of the input image of the single-enhanced super-resolution model is the same as the image resolution of the output image.

[0083] In a possible implementation manner of the second aspect, the first resolution of the first image is lower than the image resolution of the target image;

[0084] The second electronic device performs super-resolution reconstruction on the first image to obtain a target image, including:

[0085] The second electronic device performs upsampling processing on the first image to obtain a second image, and the image resolution of the second image is the same as the image resolution of the target image;

[0086] The second electronic device performs super-resolution reconstruction on the second image through a single-enhanced super-resolution model to obtain the target image, where the single-enhanced super-resolution model is a super-resolution model in which the image resolution of the input image is the same as the image resolution of the output image.

[0087] It should be noted that if the first image resolution is less than the image resolution of the target image, the second electronic device can perform upsampling processing on the first image to obtain a second image, so that the image resolution of the second image is the same as the image resolution of the target image.

[0088] Then, the second electronic device processes the second image using a single-enhanced super-resolution model to obtain a target image.

[0089] The algorithm applied for upsampling can be any one of interpolation algorithms such as the nearest neighbor method, bilinear interpolation method, and cubic interpolation method.

[0090] In a possible implementation of the second aspect, the first resolution of the first image is lower than the resolution of the target image;

[0091] The second electronic device performs super-resolution reconstruction on the first image to obtain a target image, including:

[0092] The second electronic device performs super-resolution reconstruction on the first image through a multi-enhanced super-resolution model to obtain the target image, where the multi-enhanced super-resolution model is a super-resolution model in which the image resolution of the input image is less than the image resolution of the output image.

[0093] It should be noted that if the resolution of the first image is less than the image resolution of the target image, then in addition to upsampling the first image, the second electronic device can also select a multi-enhanced super-resolution model to perform super-resolution reconstruction on the first image.

[0094] The image resolution of the input image of the multi-enhanced super-resolution model is less than the image resolution of the output image. That is to say, the multi-enhanced super-resolution model can increase the image resolution of the input image.

[0095] In a third aspect, an electronic device is provided, including:

[0096] A native data module, configured to obtain native image data, where the native image data is image data generated by an application program and not yet rendered;

[0097] A preliminary rendering module, configured to render the native image data through a first graphics rendering hardware to obtain a first image;

[0098] A first super-resolution module, configured to perform super-resolution reconstruction on the first image through a second graphics rendering hardware to obtain a target image, where the first graphics rendering hardware and the second graphics rendering hardware are different graphics rendering hardwares.

[0099] In a possible implementation of the third aspect, the first super-resolution module includes:

[0100] A first identification sub-module, configured to obtain an identification of the application program;

[0101] A first model sub-module, configured to find a target super-resolution model associated with the identification;

[0102] A first reconstruction sub-module, configured to perform super-resolution reconstruction on the first image through the first graphics rendering hardware and the found target super-resolution model to obtain a target image.

[0103] In a possible implementation manner of the third aspect, the first super-resolution module further includes:

[0104] A first general sub-module, configured to perform super-resolution reconstruction on the first image through the first graphics rendering hardware and a preset general super-resolution model to obtain a target image if no target super-resolution model associated with the identifier is found.

[0105] In a possible implementation manner of the third aspect, the first super-resolution module further includes:

[0106] A first association sub-module, configured to establish an association relationship between the identifier and the general super-resolution model.

[0107] In a possible implementation manner of the third aspect, the preliminary rendering module is specifically configured to render the native image data through the first graphics rendering hardware and a preset first image resolution to obtain a first image.

[0108] In a possible implementation manner of the third aspect, the first super-resolution module is specifically configured to perform super-resolution reconstruction on the first image through the second graphics rendering hardware and a single-fold enhanced super-resolution model to obtain a target image, where the first image resolution is the same as the image resolution of the target image, and the single-fold enhanced super-resolution model is a super-resolution model in which the image resolution of the input image is the same as the image resolution of the output image.

[0109] In another possible implementation manner of the third aspect, the first super-resolution module is specifically configured to perform super-resolution reconstruction on the first image through the second graphics rendering hardware and a multi-fold enhanced super-resolution model to obtain a target image, where the first image resolution is less than the image resolution of the target image, and the multi-fold enhanced super-resolution model is a super-resolution model in which the image resolution of the input image is less than the image resolution of the output image.

[0110] In a possible implementation manner of the third aspect, the preliminary rendering module is specifically configured to render the native image data through a graphics processing unit to obtain a first image;

[0111] Correspondingly, the first super-resolution module is specifically configured to perform super-resolution reconstruction on the first image through a neural network processing unit to obtain a target image.

[0112] In a fourth aspect, there is provided an electronic device, including:

[0113] An image receiving module, configured to receive a first image sent by a first electronic device, where the first image is an image obtained after rendering the original image data generated by the application program of the first electronic device;

[0114] A second super-resolution module, configured to perform super-resolution reconstruction on the first image to obtain a target image.

[0115] In a possible implementation manner of the fourth aspect, the second super-resolution module includes:

[0116] A second identification sub-module, configured to obtain an identification of the application program;

[0117] A second model sub-module, configured to find a target super-resolution model associated with the identification;

[0118] A second reconstruction sub-module, configured to perform super-resolution reconstruction on the first image through the found target super-resolution model to obtain a target image.

[0119] In a possible implementation manner of the fourth aspect, the second super-resolution module further includes:

[0120] A second general sub-module, configured to, if no target super-resolution model associated with the identification is found, perform super-resolution reconstruction on the first image through a preset general super-resolution model to obtain a target image.

[0121] In a possible implementation manner of the fourth aspect, the second super-resolution module further includes:

[0122] A second association module, configured to establish an association relationship between the identification and the general super-resolution model.

[0123] In a possible implementation manner of the fourth aspect, the first resolution of the first image is the same as the image resolution of the target image;

[0124] The second super-resolution module is specifically configured to perform super-resolution reconstruction on the first image through a single-fold enhanced super-resolution model to obtain the target image, where the single-fold enhanced super-resolution model is a super-resolution model in which the image resolution of the input image is the same as the image resolution of the output image.

[0125] In another possible implementation manner of the fourth aspect, the first resolution of the first image is lower than the image resolution of the target image;

[0126] The second super-resolution module includes:

[0127] An upsampling sub-module, configured to perform upsampling processing on the first image to obtain a second image, where the image resolution of the second image is the same as that of the target image;

[0128] An enhancer module, configured to perform super-resolution reconstruction on the second image by the single-fold enhanced super-resolution model in the second electronic device to obtain the target image, where the single-fold enhanced super-resolution model is a super-resolution model with the same image resolution for the input image and the output image.

[0129] In another possible implementation manner of the fourth aspect, the first resolution of the first image is lower than the resolution of the target image;

[0130] The second super-resolution module is specifically configured to perform super-resolution reconstruction on the first image by a multi-fold enhanced super-resolution model to obtain the target image, where the multi-fold enhanced super-resolution model is a super-resolution model with the image resolution of the input image less than that of the output image.

[0131] In a fifth aspect, there is provided an image processing system, where the system includes a first electronic device and a second electronic device;

[0132] The first electronic device is configured to render the native image data generated by the application program to obtain a first image, and send the first image to the second electronic device;

[0133] The second electronic device is configured to execute the image processing method mentioned in the second aspect above.

[0134] In a sixth aspect, there is provided an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the electronic device implements the steps of the method as described above.

[0135] In a seventh aspect, there is provided a computer-readable storage medium storing a computer program, which when executed by a processor causes an electronic device to implement the steps of the method as described above.

[0136] In an eighth aspect, there is provided a chip system, which may be a single chip or a chip module composed of multiple chips. The chip system includes a memory and a processor, and the processor executes the computer program stored in the memory to implement the steps of the method as described above.

[0137] The beneficial effects of the embodiments of the present application compared with the prior art are:

[0138] In the image processing method of the present application, the electronic device first renders the original image data to obtain a first image, and then performs super-resolution reconstruction on the first image to obtain the target image displayed on the screen. Compared with the method of directly rendering the original image data generated by the application into the target image finally displayed on the screen, generating the target image in the way of preliminary rendering plus super-resolution reconstruction can reduce the rendering power consumption, reduce the amount of calculation, and improve the rendering efficiency.

[0139] Moreover, the electronic device performs the preliminary rendering operation through the first graphics rendering hardware and performs the super-resolution reconstruction operation through the second graphics rendering hardware, which can make full use of the heterogeneous hardware resources in the electronic device and avoid the electronic device running high-quality products with a lower image quality due to insufficient hardware resources of a single graphics rendering hardware.

[0140] In summary, the image processing method of the present application can reduce the rendering power consumption and the amount of calculation for rendering high-quality images, and can make full use of the heterogeneous hardware resources in the electronic device, solving the problem that in the existing image solutions, when the hardware resources of the electronic device are insufficient, the electronic device can only run high-quality products with a lower image quality, and has strong usability and practicality. BRIEF DESCRIPTION OF THE DRAWINGS

[0141] Figure 1 is a schematic structural diagram of an electronic device provided by an embodiment of the present application;

[0142] Figure 2 is a software structure block diagram of an electronic device provided by an embodiment of the present application;

[0143] Figure 3 is a schematic flowchart of an image processing method provided by an embodiment of the present application;

[0144] Figure 4 is a schematic diagram of an example image provided by an embodiment of the present application;

[0145] Figure 5 is a schematic diagram of another example image provided by an embodiment of the present application;

[0146] Figure 6 is a schematic diagram of another example image provided by an embodiment of the present application;

[0147] Figure 7 is a schematic diagram of another electronic device provided by an embodiment of the present application;

[0148] Figure 8 is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0149] Figure 9 is a schematic flowchart of another image processing method provided by an embodiment of the present application;

[0150] Figure 10 It is a schematic diagram of another application scenario provided by an embodiment of the present application;

[0151] Figure 11 It is a schematic flowchart of another image processing method provided by an embodiment of the present application;

[0152] Figure 12 It is a schematic diagram of another application scenario provided by an embodiment of the present application;

[0153] Figure 13 It is a schematic diagram of another application scenario provided by an embodiment of the present application;

[0154] Figure 14 It is a schematic diagram of another application scenario provided by an embodiment of the present application;

[0155] Figure 15 It is a schematic diagram of another application scenario provided by an embodiment of the present application;

[0156] Figure 16 It is a schematic diagram of another application scenario provided by an embodiment of the present application;

[0157] Figure 17 It is a schematic flowchart of another image processing method provided by an embodiment of the present application;

[0158] Figure 18 It is a schematic diagram of another application scenario provided by an embodiment of the present application;

[0159] Figure 19 It is a schematic flowchart of another image processing method provided by an embodiment of the present application;

[0160] Figure 20 It is a schematic diagram of another application scenario provided by an embodiment of the present application;

[0161] Figure 21 It is a schematic diagram of another application scenario provided by an embodiment of the present application;

[0162] Figure 22 It is a schematic diagram of another application scenario provided by an embodiment of the present application;

[0163] Figure 23 It is a schematic diagram of another application scenario provided by an embodiment of the present application;

[0164] Figure 24 It is a schematic diagram of another application scenario provided by an embodiment of the present application;

[0165] Figure 25 It is a schematic diagram of another application scenario provided by an embodiment of the present application;

[0166] Figure 26 It is a schematic diagram of another electronic device provided by an embodiment of the present application. Specific embodiments

[0167] In the following description, for the purpose of illustration rather than limitation, specific details such as specific system architectures, technologies, etc. are set forth in order to provide a thorough understanding of the embodiments of the present application. However, those skilled in the art should understand that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to impede the description of the present application with unnecessary details.

[0168] It should be understood that when used in the specification of the present application and the appended claims, the term "comprising" indicates the presence of the described features, wholes, steps, operations, elements, and / or hardware, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, hardware, and / or their combinations.

[0169] It should also be understood that the term "and / or" used in the specification of the present application and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0170] As used in the specification of the present application and the appended claims, the term "if" may be interpreted as "when", "once", "in response to determining", or "in response to detecting" according to the context. Similarly, the phrase "if determined" or "if detecting [the described condition or event]" may be interpreted as meaning "once determined", "in response to determining", "once detecting [the described condition or event]", or "in response to detecting [the described condition or event]" according to the context.

[0171] In addition, in the description of the specification of the present application and the appended claims, the terms "first", "second", "third", etc. are only used for distinguishing descriptions and cannot be understood as indicating or implying relative importance.

[0172] The reference to "one embodiment" or "some embodiments" etc. described in the specification of the present application means that a specific feature, structure, or characteristic described in connection with that embodiment is included in one or more embodiments of the present application. Thus, the statements "in one embodiment", "in some embodiments", "in other some embodiments", "in still other embodiments", etc. that appear in different places in this specification are not necessarily all referring to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "comprising", "including", "having", and their variants all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0173] Before introducing the embodiments of the present application, some terms related to the embodiments of the present application are first explained:

[0174] Image quality refers to the quality of the picture. There are many image quality metrics for evaluating the image quality. The more common image quality metric is the image resolution. Under the condition that other image quality metrics are the same, the higher the image resolution, the higher the image quality, and the lower the image resolution, the lower the image quality.

[0175] In addition to the image resolution, the image quality metrics may also include one or more of the metrics such as clarity, sharpness, lens distortion, chromatic dispersion, resolution, color gamut range, color purity (color saturation), and color balance parameters.

[0176] Image resolution refers to the amount of information stored in the image, which can be understood as the number of pixels contained in the image. The expression of the image resolution can be "horizontal pixel number × vertical pixel number". For example, the resolution of an image is 2048×1080, indicating that each row of pixels in the image contains 2048 pixels and each column of pixels contains 1080 pixels.

[0177] Rendering in computer terms refers to the process of generating an image based on an image model. An image model is a description of a three-dimensional scene in a strictly defined language or data structure, which includes geometry, viewpoint, texture, lighting information, and rendering parameters. The rendering parameters may include the above-mentioned image quality metrics.

[0178] Currently, there are more and more products with fine pictures on the market. For example, high-quality games and videos.

[0179] These products have high requirements for the hardware resources of electronic devices. When an electronic device renders the pictures of these products, the rendering power consumption is high, and if the hardware resources of the electronic device are insufficient, then the electronic device can only run these products with a lower image quality, affecting the user experience.

[0180] For example, if a user runs a finely produced game on an electronic device with hardware resources equivalent to an RTX2080 graphics card, even if the user adjusts the image resolution to 2048×1080, the frame rate of the electronic device can remain at about 90 frames per second.

[0181] However, if a user runs a finely produced game on a mobile phone platform with a high image resolution, even if the mobile phone platform is the current high-level mobile phone platform Mali-G76, the frame rate of the electronic device can usually only remain at about 40 frames per second. If the user lowers the image resolution, the frame rate of the electronic device can usually only remain at 60 frames per second.

[0182] In response to this, an image acceleration (GPU-turbo) technology is provided. The GPU-turbo technology reconstructs the traditional GPU architecture at the system level of the electronic device, realizes the cooperation between software and hardware, and greatly improves the overall computing efficiency of the GPU. Moreover, the GPU-turbo technology can detect the similarities and differences in the image quality of adjacent frames through artificial intelligence (AI) technology, render the different parts of adjacent frames, and retain the same content of adjacent frames. In this way, GPU-turbo can save 80% of the computing, and greatly improve the rendering speed of the GPU.

[0183] However, the image rendering method of the GPU-turbo technology is the same as the traditional GPU rendering method, that is, the GPU directly renders the initial image file into the final image for on-screen display. Therefore, when using the GPU-turbo technology to run high-quality products, it will still cause a large load on the GPU of the electronic device, and the rendering power consumption is high.

[0184] In view of this, the embodiments of the present application provide an image processing method, apparatus, electronic device, and computer-readable storage medium to solve the problems of high rendering power consumption and large amount of calculation in the existing image rendering method when rendering high-quality products.

[0185] It can be understood that the steps involved in the image rendering method provided in the embodiments of the present application are only examples, not all steps are required to be executed, or not all the content in each information or message is mandatory. During use, it can be increased or decreased as needed.

[0186] In the embodiments of the present application, the same step or steps with the same function or messages can be referred to and learned from each other between different embodiments.

[0187] The business scenarios described in the embodiments of the present application are to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation to the technical solutions provided by the embodiments of the present application. Those of ordinary skill in the art know that with the evolution of the network architecture and the emergence of new business scenarios, the technical solutions provided by the embodiments of the present application are equally applicable to similar technical problems.

[0188] The electronic device described in the embodiments of the present application can be a mobile phone, a tablet computer, a handheld computer, a personal digital assistant (PDA), an augmented reality (AR) / virtual reality (VR) device, a media player, a wearable device, etc. The embodiments of the present application do not impose any restrictions on the specific form / type of the electronic device.

[0189] Exemplarily, Figure 1 A schematic structural diagram of an electronic device 100 is shown.

[0190] The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0191] It can be understood that the structure schematically shown in the embodiments of the present invention does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than those shown in the figure, or combine certain components, or split certain components, or have different component arrangements. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0192] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0193] The controller can generate operation control signals according to the instruction operation code and timing signals to complete the control of instruction fetching and execution.

[0194] A memory can also be set in the processor 110 to store instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can save the instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instruction or data again, it can directly call it from the said memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0195] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0196] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple groups of I2C buses. The processor 110 can be respectively coupled to the touch sensor 180K, charger, flashlight, camera 193, etc. through different I2C bus interfaces. For example: the processor 110 can be coupled to the touch sensor 180K through the I2C interface, enabling the processor 110 to communicate with the touch sensor 180K through the I2C bus interface to implement the touch function of the electronic device 100.

[0197] The I2S interface can be used for audio communication. In some embodiments, the processor 110 may include multiple groups of I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to enable communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 via the I2S interface to implement the function of answering a call through a Bluetooth headset.

[0198] The PCM interface can also be used for audio communication to sample, quantize, and encode analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled via the PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface to implement the function of answering a call through a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0199] The UART interface is a general-purpose serial data bus for asynchronous communication. This bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is typically used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 via the UART interface to implement the Bluetooth function. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 via the UART interface to implement the function of playing music through a Bluetooth headset.

[0200] The MIPI interface can be used to connect the processor 110 to peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to implement the shooting function of the electronic device 100. The processor 110 and the display screen 194 communicate via the DSI interface to implement the display function of the electronic device 100.

[0201] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 to the camera 193, the display screen 194, the wireless communication module 160, the audio module 170, the sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0202] The USB interface 130 is an interface that complies with the USB standard specification. Specifically, it can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the electronic device 100, and can also be used to transfer data between the electronic device 100 and peripheral devices. It can also be used to connect headphones to play audio. This interface can also be used to connect other electronic devices, such as AR devices, etc.

[0203] It can be understood that the interface connection relationship between the modules schematically shown in the embodiments of the present invention is only for illustrative purposes and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 can also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0204] The charging management module 140 is used to receive a charging input from a charger. Among them, the charger can be a wireless charger or a wired charger. In some embodiments of wired charging, the charging management module 140 can receive the charging input of the wired charger through the USB interface 130. In some embodiments of wireless charging, the charging management module 140 can receive the wireless charging input through the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also supply power to the electronic device through the power management module 141.

[0205] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives the input from the battery 142 and / or the charging management module 140 and supplies power to the processor 110, the internal memory 121, the display screen 194, the camera 193, the wireless communication module 160, etc. The power management module 141 can also be used to monitor parameters such as the battery capacity, the number of battery cycles, and the battery health status (leakage, impedance). In some other embodiments, the power management module 141 can also be provided in the processor 110. In some other embodiments, the power management module 141 and the charging management module 140 can also be provided in the same device.

[0206] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modulation and demodulation processor, and the baseband processor, etc.

[0207] The antenna 1 and the antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization rate of the antennas. For example: The antenna 1 can be multiplexed as the diversity antenna of the wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0208] The mobile communication module 150 may provide a solution for wireless communication including 2G / 3G / 4G / 5G, etc., which is applied to the electronic device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 may receive electromagnetic waves through the antenna 1, filter, amplify, etc. the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 may also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves through the antenna 1 and radiate it out. In some embodiments, at least some functional modules of the mobile communication module 150 may be disposed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be disposed in the same device.

[0209] The modulation and demodulation processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. Subsequently, the demodulator transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs a sound signal through an audio device (not limited to the speaker 170A, receiver 170B, etc.), or displays an image or video through the display screen 194. In some embodiments, the modulation and demodulation processor may be an independent device. In other embodiments, the modulation and demodulation processor may be independent of the processor 110 and be disposed in the same device as the mobile communication module 150 or other functional modules.

[0210] The wireless communication module 160 may provide solutions for wireless communications applied to the electronic device 100, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite systems (GNSSs), frequency modulation (FM), near field communication (NFC), infrared (IR), etc. The wireless communication module 160 may be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, performs frequency modulation and filtering processing on the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 may also receive signals to be sent from the processor 110, perform frequency modulation and amplification on them, and convert them into electromagnetic waves through the antenna 2 for radiation.

[0211] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, such that electronic device 100 can communicate with a network and other devices through wireless communication technologies. The wireless communication technologies may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology, etc. The GNSS may include global positioning system (GPS), global navigation satellite system (GLONASS), beidou navigation satellite system (BDS), quasi-zenith satellite system (QZSS), and / or satellite based augmentation systems (SBAS).

[0212] Electronic device 100 implements a display function through a GPU, display screen 194, and an application processor, etc. The GPU is a microprocessor for image processing, and is connected to display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

[0213] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include one or N display screens 194, where N is a positive integer greater than 1.

[0214] The electronic device 100 can implement the shooting function through the ISP, the camera 193, the video codec, the GPU, the display screen 194, and the application processor, etc.

[0215] The ISP is used to process the data fed back by the camera 193. For example, when taking a photo, the shutter is opened, and the light passes through the lens and is transmitted to the camera photosensitive element. The optical signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. The ISP can also optimize parameters such as the exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0216] The camera 193 is used to capture static images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in standard RGB, YUV, etc. formats. In some embodiments, the electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0217] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy, etc.

[0218] The video codec is used to compress or decompress digital videos. The electronic device 100 can support one or more video codecs. In this way, the electronic device 100 can play or record videos in multiple encoding formats, such as: Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0219] The NPU is a neural-network (NN) computing processor. By drawing on the structure of the biological neural network, such as the transmission mode between human brain neurons, it can quickly process the input information and can also continuously self-learn. Through the NPU, applications such as intelligent cognition of the electronic device 100 can be realized, such as: image recognition, face recognition, voice recognition, text understanding, etc.

[0220] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to achieve the data storage function. For example, files such as music and videos are saved in the external memory card.

[0221] The internal memory 121 can be used to store computer-executable program code, and the executable program code includes instructions. The internal memory 121 can include a program storage area and a data storage area. Among them, the program storage area can store the operating system, application programs required for at least one function (such as the sound playback function, image playback function, etc.). The data storage area can store the data created during the use of the electronic device 100 (such as audio data, phone book, etc.). In addition, the internal memory 121 can include high-speed random access memory and can also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121 and / or the instructions stored in the memory provided in the processor.

[0222] The electronic device 100 can implement audio functions through the audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and the application processor, etc. For example, music playback, recording, etc.

[0223] The audio module 170 is used to convert digital audio information into an analog audio signal for output, and is also used to convert an analog audio input into a digital audio signal. The audio module 170 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 170 can be disposed in the processor 110, or some functional modules of the audio module 170 can be disposed in the processor 110.

[0224] The speaker 170A, also known as a "loudspeaker", is used to convert an audio electrical signal into a sound signal. The electronic device 100 can listen to music or hands-free calls through the speaker 170A.

[0225] The receiver 170B, also known as an "earpiece", is used to convert an audio electrical signal into a sound signal. When the electronic device 100 answers a call or a voice message, the voice can be listened to by bringing the receiver 170B close to the human ear.

[0226] The microphone 170C, also known as a "microphone" or "transmitter", is used to convert a sound signal into an electrical signal. When making a call or sending a voice message, the user can speak by bringing the mouth close to the microphone 170C to input the sound signal into the microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In some other embodiments, the electronic device 100 can be provided with two microphones 170C, which can not only collect sound signals but also implement a noise reduction function. In some other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C, which can collect sound signals, reduce noise, identify the sound source, and implement a directional recording function, etc.

[0227] The headphone jack 170D is used to connect a wired headphone. The headphone jack 170D can be a USB interface 130, or a 3.5 mm open mobile terminal platform (OMTP) standard interface, or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0228] The pressure sensor 180A is used to sense pressure signals and can convert pressure signals into electrical signals. In some embodiments, the pressure sensor 180A may be disposed on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor may include at least two parallel plates having conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of the pressure based on the change in capacitance. When a touch operation acts on the display screen 194, the electronic device 100 detects the intensity of the touch operation according to the pressure sensor 180A. The electronic device 100 can also calculate the position of the touch based on the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities may correspond to different operation instructions. For example, when a touch operation with a touch operation intensity less than the first pressure threshold acts on the short message application icon, the instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the short message application icon, the instruction to create a new short message is executed.

[0229] The gyroscope sensor 180B can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (i.e., the x, y, and z axes) can be determined by the gyroscope sensor 180B. The gyroscope sensor 180B can be used for anti-shake shooting. Exemplarily, when the shutter is pressed, the gyroscope sensor 180B detects the angle of jitter of the electronic device 100, calculates the distance that the lens module needs to compensate according to the angle, and makes the lens offset the jitter of the electronic device 100 through reverse movement to achieve anti-shake. The gyroscope sensor 180B can also be used for navigation and somatosensory game scenarios.

[0230] The barometric pressure sensor 180C is used to measure barometric pressure. In some embodiments, the electronic device 100 calculates the altitude based on the barometric pressure value measured by the barometric pressure sensor 180C to assist in positioning and navigation.

[0231] The magnetic sensor 180D includes a Hall sensor. The electronic device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip leather case. In some embodiments, when the electronic device 100 is a flip phone, the electronic device 100 can detect the opening and closing of the flip according to the magnetic sensor 180D. Furthermore, according to the detected opening and closing state of the leather case or the opening and closing state of the flip, features such as automatic flip unlocking are set.

[0232] The acceleration sensor 180E can detect the magnitude of the acceleration of the electronic device 100 in various directions (generally three axes). When the electronic device 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the electronic device and is applied to applications such as horizontal and vertical screen switching and pedometers.

[0233] A distance sensor 180F for measuring distance. The electronic device 100 can measure distance through infrared or laser. In some embodiments, when shooting a scene, the electronic device 100 can use the distance sensor 180F to measure distance to achieve fast focusing.

[0234] The proximity light sensor 180G may include, for example, a light-emitting diode (LED) and a light detector, such as a photodiode. The light-emitting diode may be an infrared light-emitting diode. The electronic device 100 emits infrared light outward through the light-emitting diode. The electronic device 100 uses the photodiode to detect the infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the electronic device 100. When insufficient reflected light is detected, the electronic device 100 can determine that there is no object near the electronic device 100. The electronic device 100 can use the proximity light sensor 180G to detect that the user holds the electronic device 100 close to the ear for a call, so as to automatically turn off the screen to save power. The proximity light sensor 180G can also be used for automatic unlocking and locking of the holster mode and pocket mode.

[0235] The ambient light sensor 180L is used to sense the ambient light brightness. The electronic device 100 can adaptively adjust the brightness of the display screen 194 according to the sensed ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the electronic device 100 is in the pocket to prevent accidental touch.

[0236] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to achieve fingerprint unlocking, access application locks, fingerprint photography, fingerprint answering of incoming calls, etc.

[0237] The temperature sensor 180J is used to detect temperature. In some embodiments, the electronic device 100 executes a temperature processing strategy using the temperature detected by the temperature sensor 180J. For example, when the temperature reported by the temperature sensor 180J exceeds a threshold, the electronic device 100 reduces the performance of the processor located near the temperature sensor 180J to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is lower than another threshold, the electronic device 100 heats the battery 142 to avoid abnormal shutdown of the electronic device 100 caused by low temperature. In other embodiments, when the temperature is lower than yet another threshold, the electronic device 100 boosts the output voltage of the battery 142 to avoid abnormal shutdown caused by low temperature.

[0238] The touch sensor 180K, also known as the "touch control device". The touch sensor 180K can be disposed on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also known as the "touch control screen". The touch sensor 180K is used to detect touch operations acting thereon or nearby. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In some other embodiments, the touch sensor 180K can also be disposed on the surface of the electronic device 100, at a different position from that of the display screen 194.

[0239] The bone conduction sensor 180M can acquire vibration signals. In some embodiments, the bone conduction sensor 180M can acquire the vibration signals of the vibrating bone mass of the human vocal part. The bone conduction sensor 180M can also contact the human pulse to receive the blood pressure pulsation signal. In some embodiments, the bone conduction sensor 180M can also be disposed in the earphone to form a bone conduction earphone. The audio module 170 can parse out the voice signal based on the vibration signal of the vibrating bone mass of the human vocal part acquired by the bone conduction sensor 180M to implement the voice function. The application processor can parse the heart rate information based on the blood pressure pulsation signal acquired by the bone conduction sensor 180M to implement the heart rate detection function.

[0240] The button 190 includes a power-on button, a volume button, etc. The button 190 can be a mechanical button or a touch button. The electronic device 100 can receive button inputs to generate key signal inputs related to the user settings and function control of the electronic device 100.

[0241] The motor 191 can generate vibration prompts. The motor 191 can be used for incoming call vibration prompts and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, playing audio, etc.) can correspond to different vibration feedback effects. For touch operations acting on different regions of the display screen 194, the motor 191 can also correspond to different vibration feedback effects. Different application scenarios (such as time reminder, receiving information, alarm clock, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0242] The indicator 192 can be an indicator light and can be used to indicate the charging state, power change, and can also be used to indicate messages, missed calls, notifications, etc.

[0243] The SIM card interface 195 is used to connect to the SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to achieve contact and separation from the electronic device 100. The electronic device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, etc. Multiple cards can be inserted into the same SIM card interface 195 simultaneously. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to implement functions such as calls and data communication. In some embodiments, the electronic device 100 uses an eSIM, that is, an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

[0244] The software system of the electronic device 100 can adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservices architecture, or a cloud architecture. In the embodiments of the present invention, the Android system with a layered architecture is taken as an example to exemplarily illustrate the software structure of the electronic device 100.

[0245] Figure 2 It is the software structure block diagram of the electronic device 100 in the embodiments of the present invention.

[0246] The layered architecture divides the software into several layers, and each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer.

[0247] The application layer can include a series of application packages.

[0248] As Figure 2 shown, the application packages can include applications such as a camera, a gallery, a calendar, a call, a map, a navigation, a WLAN, a Bluetooth, music, a video, a short message, etc.

[0249] The application framework layer provides application programming interfaces (APIs) and programming frameworks for the applications in the application layer. The application framework layer includes some predefined functions.

[0250] As Figure 2 shown, the application framework layer can include a window manager, a content provider, a view system, a telephone manager, a resource manager, a notification manager, etc.

[0251] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0252] The content provider is used to store and obtain data, and make this data accessible to application programs. The data may include videos, images, audio, incoming and outgoing calls, browsing history and bookmarks, phone books, etc.

[0253] The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build application programs. The display interface can be composed of one or more views. For example, a display interface including a short message notification icon may include a view for displaying text and a view for displaying pictures.

[0254] The phone manager is used to provide the communication function of the electronic device 100. For example, the management of call states (including connection, disconnection, etc.).

[0255] The resource manager provides various resources for application programs, such as localized strings, icons, pictures, layout files, video files, etc.

[0256] The notification manager enables application programs to display notification information in the status bar. It can be used to convey notification-type messages, which can automatically disappear after a short stay without user interaction. For example, the notification manager is used to inform that the download is completed, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as the notification of a background-running application program, and can also be a notification that appears on the screen in the form of a dialogue window. For example, prompt text information in the status bar, emit a prompt sound, the electronic device vibrates, the indicator light flashes, etc.

[0257] Android Runtime includes core libraries and a virtual machine. Android runtime is responsible for the scheduling and management of the Android system.

[0258] The core libraries contain two parts: one part is the functional functions that need to be called by the Java language, and the other part is the core libraries of Android.

[0259] The application layer and the application framework layer run in the virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0260] The system library may include multiple functional modules. For example: the surface manager, Media Libraries, 3D graphics processing library (e.g., OpenGL ES), 2D graphics engine (e.g., SGL), etc.

[0261] The surface manager is used to manage the display subsystem and provides the fusion of 2D and 3D layers for multiple applications.

[0262] The media library supports the playback and recording of multiple common audio and video formats, as well as static image files, etc. The media library can support multiple audio and video coding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0263] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, synthesis, and layer processing, etc.

[0264] The 2D graphics engine is the drawing engine for 2D drawing.

[0265] The kernel layer is the layer between hardware and software. The kernel layer at least includes a display driver, a camera driver, an audio driver, and a sensor driver.

[0266] Next, in combination with the capture and photographing scenario, the working processes of the software and hardware of the electronic device 100 will be described by way of example.

[0267] When the touch sensor 180K receives a touch operation, the corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into a raw input event (including information such as touch coordinates and the timestamp of the touch operation). The raw input event is stored in the kernel layer. The application framework layer obtains the raw input event from the kernel layer and identifies the control corresponding to the input event. Taking the touch operation as a touch click operation and the control corresponding to the click operation as the control of the camera application icon as an example, the camera application calls the interface of the application framework layer to start the camera application, and then starts the camera driver by calling the kernel layer, and captures a static image or video through the camera 193.

[0268] Next, from the perspective of the first electronic device, an image processing method provided in this embodiment will be described. Please refer to Figure 3 The flowchart of the image processing method shown, the method includes:

[0269] S301. The first electronic device renders the native image data through the first graphics rendering hardware to obtain a first image;

[0270] When a user is using an application on a first electronic device, the application passes the native image data to the first graphics rendering hardware of the first electronic device for rendering. Native image data refers to an image file generated by the application without being rendered.

[0271] Rendering refers to the process of converting the image data stored in the first electronic device into visible pixels through techniques such as rasterization.

[0272] In the image processing method of this embodiment, the first electronic device can obtain a target image with target image quality metrics through image super-resolution reconstruction. However, the first electronic device cannot directly perform super-resolution reconstruction on the native image data.

[0273] Therefore, the first electronic device needs to perform preliminary rendering on the native image data to obtain a first image that can be subjected to super-resolution reconstruction.

[0274] During the preliminary rendering process, the first electronic device can call the first graphics rendering hardware to render the native image data with a first image quality metric to obtain a first image.

[0275] When the first graphics rendering hardware performs preliminary rendering on the native image data with a first image quality metric, it can render the native image data with a first image resolution, where the first image resolution is lower than the preset image resolution; or, the first graphics rendering hardware can also adjust other image quality metrics so that the first image quality metric is lower than the preset image quality metric, and then render the native image data.

[0276] The above first image quality metric can be a specific image quality metric, or the above first image quality metric can be a set of multiple image quality metrics. When the above first image quality metric is a set of multiple image quality metrics, the above first image quality metric being lower than the preset image quality metric can be understood as some or all of the image quality metrics in the first image quality metric being lower than the preset image quality metric.

[0277] For example, as Figure 4 and Figure 5 shown in the example images, when other image quality metrics except for the image resolution are the same, Figure 4 the image resolution of Figure 5 is lower than the image resolution of Figure 4 then it can be considered that the image quality of Figure 5 is lower than the image quality of Figure 5 and Figure 6 shown in the example images, when Figure 5 and Figure 6 have the same image resolution, since Figure 6 is less clear than Figure 5 it can also be considered that the image quality of Figure 6 is lower thanFigure 5 image quality

[0278] Since when the first graphics rendering hardware renders an image, the higher the image quality index of the rendered image, the more hardware resources the first graphics rendering hardware occupies and the higher the rendering power consumption. Therefore, when the first graphics rendering hardware preliminarily renders the native image data with a first image quality index lower than the preset image quality index, the hardware resources occupied by the first graphics rendering hardware during rendering can be reduced, and the rendering power consumption can be reduced.

[0279] The above first graphics rendering hardware can be selected according to the actual situation. In some embodiments, the first graphics rendering hardware can be one or more of a central processing unit (CPU), a graphics processing unit (GPU), and a neural-network processing unit (NPU). For example, when the first electronic device uses GPU-turbo technology to preliminarily render the native image data, the above first graphics rendering hardware can be a combination of a CPU and a GPU.

[0280] S302. The first electronic device performs super-resolution reconstruction on the first image through a second graphics rendering hardware to obtain a target image.

[0281] In S301, the first electronic device preliminarily renders the native image data of the application through the first graphics rendering hardware to obtain a first image. However, the image quality of the first image is low and it is difficult to meet the user's requirements for the product image quality.

[0282] Super-resolution reconstruction refers to applying AI technology to map a low-resolution image to a high-resolution one in order to enhance the image quality.

[0283] After the first graphics rendering hardware obtains the first image, the first electronic device can use the second graphics rendering hardware to perform super-resolution reconstruction on the above first image.

[0284] When the first electronic device performs super-resolution reconstruction on the first image, the above first image can be input into a trained super-resolution model, and the super-resolution model enhances the image quality of the first image to obtain a target image.

[0285] The type of the above super-resolution model can be selected according to the actual situation. For example, the above super-resolution model can be any one of a super-resolution convolutional neural network model (SRCNN model), a fast super-resolution convolutional neural network model, an efficient sub-pixel convolutional neural network model (ESPCN model), a deeply-recursive convolutional network model (DRCN model), and a very deep network for super-resolution (VSDR model), etc.

[0286] Moreover, if the first electronic device takes a long time to process a single-frame image, it will cause the application to be unable to maintain a certain display frame rate, resulting in screen stuttering on the first electronic device. Therefore, when developers select a super-resolution model, they should limit the scale of the super-resolution model so that the single-frame running time of the super-resolution model meets the requirement of the single-frame time of the display frame rate of the application. For example, assuming that the display frame rate of the application is 90 frames per second, the single-frame time of the display frame rate of the application is 1 / 90 second. At this time, the scale of the super-resolution model should be limited so that the single-frame running time of the super-resolution model is less than 1 / 90 second, thereby ensuring that the display frame rate of the application is kept at 90 frames per second as much as possible and reducing the occurrence of screen stuttering.

[0287] When training the above super-resolution model, the first electronic device can obtain at least one set of image sample pairs and train the super-resolution model with the image sample pairs. An image sample pair refers to a pair of sample images. Each set of image sample pairs includes a first sample image and a second sample image. The content of the first sample image and the second sample image is the same, but the image quality of the first sample image is lower than that of the second sample image.

[0288] When the first electronic device uses the image sample pairs to train the super-resolution model, it can input the first sample image in the image sample pairs into the super-resolution model to obtain a first output image.

[0289] Then, the first electronic device calculates a loss value based on the first output image, the second sample image, and a preset loss function, and updates the super-resolution model according to the loss value and a preset network update algorithm.

[0290] After the super-resolution model is updated, return to the previous step and use the image samples to train the super-resolution model in a loop until the number of loops reaches the preset number threshold or the loss value is less than the preset loss threshold.

[0291] The acquisition method of the above image sample pairs can be selected according to the actual situation. For example, the above image sample pairs can be the original generated image dataset, or the above image sample pairs can also be obtained by degrading high-quality images into low-quality images to obtain paired image sample pairs.

[0292] Moreover, the source of the image sample pairs will have a certain impact on the performance of the super-resolution model. If developers want to train a general super-resolution model, when sampling image sample pairs, developers can collect samples without specific targeting to obtain general image sample pairs.

[0293] At this time, the super-resolution model trained by the first electronic device using general image sample pairs can obtain a general super-resolution model. The general super-resolution model has high applicability and can be applied to many application scenarios. However, the image quality optimization ability of the general super-resolution model is limited, and it is difficult to optimize the image quality of each application scenario to a high degree.

[0294] Therefore, if developers want the super-resolution model to be able to optimize the image quality of a certain product or a certain type of product to a high degree, when sampling image sample pairs, developers should only collect image sample pairs related to the product or the type of product to obtain specific image sample pairs.

[0295] At this time, the first electronic device uses specific image sample pairs to train the super-resolution model, and a specific super-resolution model for a certain product or a certain type of product can be obtained. The applicability of the specific super-resolution model is poor and can only be applied to specific products. However, the image quality optimization ability of the specific super-resolution model is high, and it can optimize the image quality of specific products to a high degree.

[0296] Taking game applications as an example. When developers want to train a general game super-resolution model, developers can obtain image sample pairs from various game applications. For example, developers can obtain general game image sample pairs from the same type or different types of game applications such as "Arena of Valor" (Chinese translation: "Legend of the Conquerors"), "PUBG Mobile" (Chinese translation: "PUBG"), "Carrot Fantasy" (Chinese translation: "Carrot Defense"), "Plants vs. Zombies" (Chinese translation: "Plants vs. Zombies"), "Minecraft" (Chinese translation: "Minecraft"), "Life After" (Chinese translation: "Survival After"), etc.

[0297] Then, the first electronic device uses the above-mentioned general game image samples to train a super-resolution model, so that the trained super-resolution model can be applicable to various different game applications.

[0298] When a developer wants to train a targeted super-resolution model for a popular game application, the developer can only obtain the images of the game application as specific game image samples. For example, currently "Arena of Valor" is a popular game with a large number of players. To enable players to have a smoother gaming experience when playing "Arena of Valor", the developer can train a super-resolution model for "Arena of Valor".

[0299] When the developer is training a super-resolution model for "Arena of Valor", only the game images within "Arena of Valor" should be used as specific game image samples. The above-mentioned game images can be character images, terrain images, skill images, etc. within "Arena of Valor".

[0300] Then, the first electronic device trains the super-resolution model according to the above-mentioned specific game image samples, so that the trained super-resolution model can specifically enhance the image quality of the game "Arena of Valor".

[0301] When multiple trained super-resolution models are set in the first electronic device, the first electronic device can detect the identifier of the application when the user starts the application, and select the corresponding super-resolution model according to the identifier of the application to perform the operation in step S302 above.

[0302] If the first electronic device does not detect the super-resolution model corresponding to the identifier of the application, the first electronic device can call the general super-resolution model to perform the steps in step S302 above. Moreover, when multiple general super-resolution models are set in the first electronic device, the first electronic device can also establish an association relationship between the general super-resolution model and the identifier of the above-mentioned application after performing super-resolution reconstruction on the first image using the general super-resolution model, so that when the first electronic device processes the first image of the application next time, it can find the same general super-resolution model to process the first image according to the foregoing association relationship.

[0303] In some embodiments, the identifier of the above application may be the package name (packname) of the application. For example, after the first electronic device trains a super-resolution model for "Arena of Valor", it can associate the super-resolution model with the package name of "Arena of Valor". When "Arena of Valor" is launched or awakened, the first electronic device obtains the package name of "Arena of Valor", looks up the corresponding super-resolution model according to the package name, and uses the corresponding super-resolution model to process the image of "Arena of Valor".

[0304] Alternatively, in some other embodiments, the identifier of the application can also be user-defined. For example, the user defines the identifier of "Arena of Valor" as 0010. Then, the first electronic device associates "0010" with "Arena of Valor" and the super-resolution model for "Arena of Valor". When "Arena of Valor" is launched or awakened, the first electronic device looks up the corresponding identifier of "Arena of Valor" and obtains the identifier "0010" of "Arena of Valor". Then, the first electronic device looks up the corresponding super-resolution model according to the identifier "0010" and uses the corresponding super-resolution model to process the image of "Arena of Valor".

[0305] In addition, the above super-resolution model can be a single-fold enhanced super-resolution model, or the above super-resolution model can also be a multi-fold enhanced super-resolution model. A single-fold enhanced super-resolution model is a super-resolution model in which the image resolution of the input image is the same as that of the output image. A multi-fold enhanced super-resolution model is a super-resolution model in which the image resolution of the input image is smaller than that of the output image.

[0306] When the above super-resolution model is a single-fold enhanced super-resolution model, the image resolution of the first image is the same as that of the target image. The single-fold enhanced super-resolution model enhances the image quality of the first image by improving other image quality metrics of the first image to obtain the target image.

[0307] When the above super-resolution model is a multi-fold enhanced super-resolution model, the image resolution of the first image is smaller than that of the target image. When the first graphics rendering hardware of the first electronic device performs preliminary rendering on the native image data of the application, it can render at a smaller image resolution, thereby reducing the hardware resources occupied by the first graphics rendering hardware during the rendering process and reducing the rendering power consumption.

[0308] After the first graphics rendering hardware renders the first image with a small image resolution, the first electronic device then enhances the image quality of the first image through a multi-fold enhanced super-resolution model according to the target resolution configured by the user, adapts the image resolution of the first image to the target image resolution, and obtains the target image.

[0309] It should be noted that when the first electronic device performs the above super-resolution reconstruction operation, the second graphics rendering hardware that runs the above super-resolution model can be set according to the actual situation. In some embodiments, the first electronic device can run the above super-resolution model on the CPU and enhance the image quality of the first image through the CPU; in other embodiments, the first electronic device can run the above super-resolution model on the GPU and enhance the image quality of the first image through the GPU; or, the first electronic device can run the above super-resolution model on the NPU and enhance the image quality of the first image through the NPU. This application does not limit the hardware in the first electronic device that performs the above super-resolution reconstruction operation herein.

[0310] To better illustrate the image processing method provided in the embodiments of the present application, the following describes it in combination with specific scenarios.

[0311] Figure 7 It is a schematic diagram of the first electronic device applicable to Scenario 1, Scenario 2, Scenario 3, and Scenario 4 provided in this embodiment. As Figure 7 shown, a GPU 701 and an NPU 702 can be set in the first electronic device. In the NPU 702, a general game super-resolution model and a super-resolution model for Game A are preset, and the super-resolution model for Game A is associated with the application identifier of Game A.

[0312] Scenario 1:

[0313] As shown in 8, multiple icons can be set on the main page of the first electronic device, including icons such as "Clock", "Calendar", "Game B", "Memo", "Camera", "Contacts", "Phone", "Messages", etc. One icon represents an application program.

[0314] The user clicks on the icon of Game B on the first electronic device, and the first electronic device responds to the user's click operation and starts Game B.

[0315] As Figure 9 shown, after Game B is started, the application program of Game B sends native image data frame by frame to the GPU 701.

[0316] The GPU 701 renders the above native image data frame by frame, obtains the first image corresponding to each frame of native image data, and sends the first image frame by frame to the NPU 702.

[0317] After receiving the first image, NPU 702 obtains the application identifier of game B. If no super-resolution model corresponding to game B is found according to the application identifier of game B, a super-resolution model of a general game class is selected as the target super-resolution model.

[0318] NPU 702 inputs the first image into the target super-resolution model frame by frame, enhances the image quality of the first image through the target super-resolution model, obtains the target image corresponding to each frame of the first image, and sends the target image to the display screen of the first electronic device frame by frame.

[0319] like Figure 10 As shown, after the display screen acquires the target image, the target image is displayed on the screen, so that the display screen displays the screen of game B.

[0320] In addition, if only one general game-type super-resolution model is set in the first electronic device, when the user triggers game B again, the first electronic device still uses the same general game-type super-resolution model to process the image of game B.

[0321] If there are multiple super-resolution models of general game types in the first electronic device, when the user triggers game B again, the first electronic device can randomly select a super-resolution model from the multiple super-resolution models of general game types to process the image of game B.

[0322] Alternatively, if there are multiple super-resolution models of general game categories in the first electronic device, the first electronic device may also establish an association relationship between the application identifier of game B and the super-resolution model of the general game category after first using the super-resolution model of the general game category to process the image of game B. When the user triggers game B again, the first electronic device may obtain the application identifier of game B, find the super-resolution model of the same general game category according to the application identifier of game B, and process the image of game B.

[0323] Scenario 2:

[0324] like Figure 11 As shown, a plurality of icons may be provided on the main page of the first electronic device, including icons such as "clock", "calendar", "game A", "memo", "camera", "address book", "phone", "information", etc. One icon represents an application.

[0325] like Figure 9 As shown, the user clicks on the icon of game A on the first electronic device, and the first electronic device starts game A in response to the user's clicking operation.

[0326] After game A is started, the application of game A sends native image data to GPU 701 frame by frame.

[0327] The GPU 701 renders the native image data frame by frame to obtain a first image corresponding to each frame of the native image data, and sends the first image to the NPU 702 frame by frame.

[0328] After receiving the first image, NPU 702 obtains the application identifier of game A, finds the super-resolution model for game A according to the application identifier of game A, and selects the super-resolution model for game A as the target super-resolution model.

[0329] NPU 702 inputs the first image into the target super-resolution model frame by frame, enhances the image quality of the first image through the target super-resolution model, obtains the target image corresponding to each frame of the first image, and sends the target image to the display screen of the first electronic device frame by frame.

[0330] like Figure 12 As shown, after the display screen acquires the target image, the target image is displayed on the screen, so that the display screen displays the screen of game A.

[0331] Scenario 3:

[0332] like Figure 11 As shown, the user clicks on the icon of game A on the first electronic device, and the first electronic device starts game A in response to the user's clicking operation.

[0333] like Figure 13 and 14 As shown, after game A is started, the application of game A sends native image data 1301 frame by frame to GPU 701. The image resolution of native image data 1301 is 480×360.

[0334] The GPU 701 renders the native image data 1301 frame by frame to obtain a first image 1302 corresponding to each frame of the native image data 1301. The image resolution of the first image 1302 is 1920×1080. The GPU 701 sends the first image 1302 after preliminary rendering to the NPU 702 frame by frame.

[0335] After receiving the first image 1302, the NPU 702 obtains the application identifier of game A, finds the super-resolution model for game A according to the application identifier of game A, and selects the super-resolution model for game A as the target super-resolution model. The target super-resolution model is a single-enhancement super-resolution model, and the target resolution of the output image of the target super-resolution model is 1920×1080.

[0336] The NPU 702 inputs the first image 1302 into the target super-resolution model frame by frame, and enhances the image quality of the first image 1302 through the target super-resolution model to obtain a target image 1303 with an image resolution of 1920×1080.

[0337] As Figure 14 shown, the image resolution of the target image 1303 is the same as that of the first image 1302, but the clarity of the target image 1303 is higher than that of the first image 1302, and the picture quality of the target image 1303 is higher than that of the first image 1302.

[0338] After the NPU 702 obtains the target image 1303, it sends the target image 1303 to the display screen of the first electronic device frame by frame.

[0339] Scenario 4:

[0340] As Figure 11 shown, the user clicks on the icon of Game A on the first electronic device, and the first electronic device responds to the user's click operation and launches Game A.

[0341] As Figure 15 and Figure 16 shown, after Game A is launched, the application program of Game A sends the native image data 1501 to the GPU 701 frame by frame. The image resolution of the native image data 1501 is 480×360.

[0342] The GPU 701 renders the above native image data 1501 frame by frame to obtain the first image 1502 corresponding to each frame of the native image data. The image resolution of the first image 1502 is 480×360. The GPU 701 sends the preliminarily rendered first image 1502 to the NPU 702 frame by frame.

[0343] After the NPU 702 receives the first image 1502, it obtains the application identifier of Game A, finds the super-resolution model for Game A according to the application identifier of Game A, and selects the super-resolution model for Game A as the target super-resolution model. The target super-resolution model is a multi-fold enhanced super-resolution model, and the target resolution of the output image of the target super-resolution model is 1920×1080.

[0344] The NPU 702 inputs the first image 1502 into the target super-resolution model frame by frame, enhances the picture quality of the first image 1502 through the target super-resolution model, and adapts the resolution of the first image 1502 to obtain a target image 1503 with an image resolution of 1920×1080.

[0345] As Figure 16 shown, the clarity of the target image 1503 is the same as that of the first image 1502, but the image resolution of the target image 1503 is higher than that of the first image 1502, and the picture quality of the target image 1503 is higher than that of the first image 1502.

[0346] After the NPU 702 obtains the target image 1503, it sends the target image 1503 to the display screen of the first electronic device for on-screen display frame by frame.

[0347] In summary, in the image processing method provided in this embodiment, the first electronic device first performs preliminary rendering on the native image data of the application to obtain a first image. Then, the first electronic device performs super-resolution reconstruction on the first image through a super-resolution model to improve the image quality of the first image and obtain a target image for on-screen display. Compared with the current solution of directly rendering the native image data to obtain a high-quality image, the image processing method provided in this embodiment can reduce the requirements for the hardware resources of the first electronic device and reduce the rendering power consumption, solving the problems of high rendering power consumption and large computational volume in the existing image processing methods when rendering high-quality products.

[0348] In addition, the above-mentioned preliminary rendering step and super-resolution reconstruction step can be executed by the same graphics rendering hardware in the first electronic device. For example, in some embodiments, both the above-mentioned preliminary rendering step and super-resolution reconstruction step can be executed by the GPU in the first electronic device; in other embodiments, both the above-mentioned preliminary rendering step and super-resolution reconstruction step can be executed by the NPU in the first electronic device.

[0349] Alternatively, the above-mentioned preliminary rendering step and super-resolution reconstruction step can also be executed by different graphics rendering hardware in the first electronic device, that is, the above-mentioned first graphics rendering hardware and second graphics rendering hardware are different graphics rendering hardware. For example, in some embodiments, the above-mentioned preliminary rendering step can be executed by the GPU of the first electronic device, and the above-mentioned super-resolution reconstruction step can be executed by the NPU of the first electronic device; in other embodiments, the above-mentioned preliminary rendering step can be executed by the CPU of the first electronic device, and the above-mentioned super-resolution reconstruction step can be executed by the NPU of the first electronic device.

[0350] When the above-mentioned preliminary rendering step and super-resolution reconstruction step are executed by different hardware, it is possible to make full use of various heterogeneous hardware resources in the first electronic device, reduce the requirements for the hardware resources of the first electronic device, and better improve the image quality under the condition of limited hardware resources.

[0351] When performing super-resolution reconstruction, the super-resolution model selected by the first electronic device can be a single-fold enhanced super-resolution model, or the super-resolution model selected by the first electronic device can also be a multi-fold enhanced super-resolution model. When the super-resolution model selected by the first electronic device is a multi-fold enhanced super-resolution model, it is possible to reduce the hardware resources occupied by the graphics rendering hardware during the preliminary rendering process and the rendering power consumption of the preliminary rendering.

[0352] In addition, the above super-resolution model can be a general super-resolution model, or the above super-resolution model can also be a specific super-resolution model for a certain application or a certain type of application. The applicable range of the specific super-resolution model is smaller than that of the general super-resolution model, but usually the image quality enhancement effect of the specific super-resolution model is better than that of the general super-resolution model.

[0353] The above is an image processing method provided by an embodiment of the present application. Next, from the perspectives of the first electronic device and the designated device, another image processing method provided by this embodiment will be described. Please refer to Figure 8 the flowchart of the image processing method shown in the figure. The method includes:

[0354] S1701. The first electronic device renders the original image data to obtain a first image;

[0355] In this embodiment, a method for collaborative image rendering by multiple electronic devices will be introduced.

[0356] When the first electronic device interacts with other electronic devices in a multi-screen manner, the first electronic device may need to project the display screen of this device to other electronic devices.

[0357] For example, when a user is playing a game on a mobile phone, the user may feel that the screen of the mobile phone is too small and the visual effect is not good. At this time, if the user has a smart TV, the user can control the mobile phone to interact with the smart TV in a multi-screen manner, establish a communication connection between the mobile phone and the smart TV, and project the game screen on the mobile phone to the smart TV for display, so that the user can watch the game screen through the smart TV and obtain a better visual experience.

[0358] In the current screen mirroring solution, the first electronic device on the screen mirroring side needs to independently complete the image rendering work, and then transfer the rendered image to the first electronic device on the receiving side. After that, the first electronic device on the receiving side will perform resolution adaptation on the rendered image and display the image with the adapted resolution on the screen.

[0359] Since in the current screen mirroring solution, the image rendering work is completely processed by the first electronic device on the screen mirroring side, this will consume a large amount of hardware resources of the electronic device on the screen mirroring side, with high rendering power consumption, and the hardware resources of multiple electronic devices within the same local area network are not fully utilized.

[0360] Moreover, usually screen mirroring technology is used to project the display screen of a small-screen device to the screen of a large-screen device. At this time, an image with a low image resolution is difficult to adapt to a display device with a high image resolution, resulting in a poor user experience.

[0361] To this end, in the image processing method provided in this embodiment, when the user starts an application on the first electronic device and enables the multi-screen interaction function, the first electronic device can perform preliminary rendering on the native image data generated by the application to obtain a first image.

[0362] The process of preliminary rendering can refer to the description of step S301 in the previous embodiment, which will not be repeated here.

[0363] S1702. The first electronic device sends the first image to a specified device to instruct the specified device to perform super-resolution reconstruction on the first image to obtain a target image.

[0364] The specified device (i.e., the aforementioned second electronic device) is selected by the user and is another electronic device in the same local area network as the above-mentioned first electronic device.

[0365] For example, as Figure 18 shown, when the user hopes to project the game screen of the mobile phone onto the smart TV, the user can turn on the "wireless screen mirroring" function of the mobile phone. After the user turns on the "wireless screen mirroring" function of the mobile phone, the mobile phone starts to search for available electronic devices in the same local area network. The mobile phone searches for electronic device 1, electronic device 2, and electronic device 3. When the mobile phone detects the user's click operation on electronic device 1, it means that electronic device 1 is selected by the user, and the mobile phone determines electronic device 1 as the specified device.

[0366] After obtaining the first image, the first electronic device sends the first image to the specified device. The above-mentioned specified device is the electronic device to be screen-mirrored. The number of specified devices can be one, or the number of specified devices can also be multiple.

[0367] For example, please refer to Figure 18 , assuming that electronic device 1 is a smart TV and electronic device 2 is a computer. When the user operates the mobile phone, if the user hopes to project the game screen of the mobile phone onto the smart TV, then the mobile phone is the first electronic device on the screen-mirroring side, and the smart TV is the electronic device on the screen-mirrored side (i.e., the specified device). At this time, the user can turn on the "wireless screen mirroring" function of the mobile phone. After the user turns on the "wireless screen mirroring" function of the mobile phone, the mobile phone starts to search for available electronic devices in the same local area network. The mobile phone searches for electronic device 1, electronic device 2, and electronic device 3. Then the user clicks on electronic device 1, and the mobile phone responds to the user's operation and sets electronic device 1 as the specified device, and the number of specified devices is 1.

[0368] When a user operates a mobile phone, if the user wishes to simultaneously cast the game screen of the mobile phone to a smart TV and a computer, the mobile phone is the first electronic device as the casting party, and the smart TV and the computer are the electronic devices (i.e., designated devices) as the cast-to parties. At this time, the user can turn on the "Wireless Screen Mirroring" function of the mobile phone. After the user turns on the "Wireless Screen Mirroring" function of the mobile phone, the mobile phone starts to search for available first electronic devices within the same local area network. The mobile phone searches for Electronic Device 1, Electronic Device 2, and Electronic Device 3. Then the user clicks on Electronic Device 1 and Electronic Device 2, and in response to the user's operation, the mobile phone sets Electronic Device 1 and Electronic Device 2 as designated devices, and the number of designated devices is 2.

[0369] After receiving the first image, the designated device inputs the first image into a trained super-resolution model, performs super-resolution reconstruction on the first image, and obtains and displays the target image.

[0370] It can be understood that in the image processing method of this embodiment, the first electronic device can not only use the local hardware resources to render the image, but also utilize the hardware resources of the designated device to optimize the image quality, making full use of the hardware resources of multiple electronic devices within the same local area network, thereby reducing the load on the local hardware resources when the first electronic device renders high-quality images and reducing the rendering power consumption of the first electronic device when rendering high-quality images.

[0371] The super-resolution model on the designated device can be a single-fold enhanced super-resolution model, or the super-resolution model on the designated device can also be a multi-fold enhanced super-resolution model. The single-fold enhanced super-resolution model is a super-resolution model in which the image resolution of the input image is the same as the image resolution of the output image. The multi-fold enhanced super-resolution model is a super-resolution model in which the image resolution of the input image is less than the image resolution of the output image.

[0372] In some embodiments, when the super-resolution model on the designated device is a single-fold enhanced super-resolution model, the first electronic device can obtain the target image resolution configured on the designated device. The target image resolution is the image resolution of the screen display image set on the designated device.

[0373] The first electronic device renders the native image data generated by the application according to the target image resolution to obtain the first image. At this time, the image resolution of the first image is the target image resolution.

[0374] Then, the first electronic device transfers the first image to the designated device. The designated device inputs the first image into the single-fold enhanced super-resolution model, improves the image quality of the first image through the single-fold enhanced super-resolution model, and obtains and displays the target image on the screen. The image resolution of the target image is the target image resolution.

[0375] In some other embodiments, when the super-resolution model on the specified device is a single-fold enhanced super-resolution model, the first electronic device can also directly render the native image data generated by the application according to the first image resolution configured by the user to obtain the first image. At this time, the image resolution of the first image is the first image resolution.

[0376] Then, the first electronic device transfers the first image to the specified device. Since the super-resolution model on the specified device is a single-fold enhanced super-resolution model and cannot adapt the resolution of the first image to the target image resolution. The target image resolution is the image resolution set for the on-screen display of the specified device. Therefore, after obtaining the first image, the specified device can perform upsampling processing on the first image to adapt the resolution of the first image to the target image resolution configured on the specified device to obtain the second image. The image resolution of the second image is the target image resolution.

[0377] The algorithm applied for upsampling can be any one of interpolation algorithms such as the nearest neighbor method, bilinear interpolation method, cubic interpolation method, etc. The specified device performs upsampling processing on the first image through a preset upsampling algorithm to obtain the second image.

[0378] After that, the specified device inputs the second image into the single-fold enhanced super-resolution network to enhance the image quality of the second image through the single-fold enhanced super-resolution network to obtain the target image. The image resolution of the target image is the target image resolution.

[0379] In some other embodiments, when the super-resolution model on the specified device is a multi-fold enhanced super-resolution model, the first electronic device can render the native image data generated by the application according to the first image resolution configured by the user to obtain the first image. At this time, the image resolution of the first image is the first image resolution.

[0380] Then, the first electronic device transfers the first image to the specified device. The specified device inputs the first image into the multi-fold enhanced super-resolution network to enhance the image quality of the first image through the multi-fold enhanced super-resolution network and adapt the resolution of the first image to the target image resolution configured on the specified device to obtain the target image. The image resolution of the target image is the target image resolution.

[0381] In addition, the super-resolution model on the specified device can be a general super-resolution model, or the above super-resolution model can also be a specific super-resolution model trained for a certain product or a certain type of product. The training methods of the above general super-resolution model and specific super-resolution model can refer to the description of the previous embodiment and will not be elaborated here.

[0382] When multiple trained super-resolution models are set on a specified device, the specified device can, after receiving a first image, obtain the identifier of the application corresponding to the first image, and select a corresponding super-resolution model according to the identifier of the application to perform super-resolution reconstruction processing on the first image.

[0383] If the specified device fails to detect the super-resolution model corresponding to the identifier of the application, the specified device can call a general super-resolution model to perform super-resolution reconstruction processing. Moreover, when multiple general super-resolution models are set in the specified device, the specified device can also establish an association relationship between the general super-resolution model and the identifier of the application after using the general super-resolution model to perform super-resolution reconstruction on the first image, so that when the specified device processes the first image of the application next time, it can find the same general super-resolution model to process the first image according to the foregoing association relationship.

[0384] It should be noted that the first electronic device can select a suitable graphics rendering hardware according to the actual situation to perform preliminary rendering on the native image data to obtain a first image. The foregoing graphics rendering hardware can be one or more of a central processing unit (CPU), a graphics processing unit (GPU), and a neural-network processing unit (NPU). For example, when the first electronic device uses GPU-turbo technology to perform preliminary rendering on the native image data, the foregoing graphics rendering hardware can be a combination of a CPU and a GPU.

[0385] The specified device can select suitable hardware according to the actual situation to run the foregoing super-resolution model. In some embodiments, the specified device can run the foregoing super-resolution model on a CPU to enhance the image quality of the first image through the CPU; in other embodiments, the specified device can run the foregoing super-resolution model on a GPU to enhance the image quality of the first image through the GPU; or, the specified device can run the foregoing super-resolution model on an NPU to enhance the image quality of the first image through the NPU. This application does not limit the hardware for performing the foregoing super-resolution reconstruction operation in the specified device.

[0386] To better illustrate the image processing method provided in the embodiments of the present application, the following describes it in combination with a specific scenario.

[0387] In the following application scenarios, the first electronic device (i.e., the electronic device of the screen mirroring party) can be a mobile phone, and a GPU can be set in the mobile phone. The designated device (i.e., the electronic device of the screen mirroring recipient) can be a smart TV and a computer, and an NPU can be set in the smart TV and the computer. In the NPU, a super-resolution model for general games and a super-resolution model for Game A are preset, and the super-resolution model for Game A is associated with the application identifier of Game A.

[0388] Scenario Five:

[0389] The user operates the mobile phone to mirror the display screen of the mobile phone to the smart TV.

[0390] As Figure 8 and Figure 19 shown, the user clicks on the icon of Game B on the mobile phone, and the mobile phone responds to the user's click operation and launches Game B.

[0391] After Game B is launched, the application program of Game B sends the original image data frame by frame to the GPU of the mobile phone.

[0392] The GPU of the mobile phone renders the above original image data frame by frame to obtain the first image corresponding to each frame of the original image data, and sends the first image frame by frame to the smart TV through the wireless communication module of the mobile phone.

[0393] After the smart TV receives the first image, it obtains the application identifier corresponding to the first image. Since no super-resolution model corresponding to Game B is found according to the application identifier of Game B, the general game super-resolution model is selected as the target super-resolution model.

[0394] The NPU of the smart TV inputs the first image into the target super-resolution model frame by frame, enhances the image quality of the first image through the target super-resolution model to obtain the target image corresponding to each frame of the first image, and sends the target image frame by frame to the display screen of the smart TV for on-screen display.

[0395] Scenario Six:

[0396] The user operates the mobile phone to mirror the display screen of the mobile phone to the smart TV.

[0397] As Figure 9 and Figure 19 shown, the user clicks on the icon of Game A on the mobile phone, and the mobile phone responds to the user's click operation and launches Game A.

[0398] After Game A is launched, the application program of Game A sends the original image data frame by frame to the GPU of the mobile phone.

[0399] The GPU of the mobile phone renders the above original image data frame by frame to obtain the first image corresponding to each frame of the original image data, and sends the first image to the smart TV through the wireless communication module of the mobile phone.

[0400] After the smart TV receives the first image, it obtains the application identifier corresponding to the first image, and finds the super-resolution model for Game A based on the application identifier of Game A. Then, it selects the super-resolution model for Game A as the target super-resolution model.

[0401] The NPU of the smart TV inputs the first image into the target super-resolution model frame by frame, enhances the image quality of the first image through the target super-resolution model, obtains the target image corresponding to each frame of the first image, and sends the target image to the display screen of the smart TV for on-screen display frame by frame.

[0402] Scenario Seven:

[0403] Please refer to Figure 20 and Figure 21 , the user operates the mobile phone 2001 to project the display screen of the mobile phone 2001 to the smart TV 2002 and the computer 2003.

[0404] The user clicks on the icon of Game A on the mobile phone 2001. In response to the user's click operation, the mobile phone 2001 starts Game A.

[0405] After Game A is started, the application program of Game A sends the native image data 20011 to the GPU of the mobile phone 2001 frame by frame. The image resolution of the native image data 20011 is 480×360.

[0406] Moreover, the mobile phone 2001 performs data interaction with the smart TV 2002 and the computer 2003 through the wireless communication module, and obtains the image resolution 1920×1080 configured on the smart TV 2002 and the image resolution 2560×1440 configured on the computer 2003.

[0407] The mobile phone 2001 renders the above native image data 20011 frame by frame according to the image resolution 1920×1080 configured on the smart TV 2002, and obtains the image 20012 with a resolution of 1920×1080.

[0408] At the same time, the mobile phone 2001 renders the above native image data 20011 frame by frame according to the image resolution 2560×1440 configured on the computer 2003, and obtains the image 20013 with a resolution of 2560×1440.

[0409] The mobile phone 2001 sends the image 20012 to the smart TV 2002 through the wireless communication module, and sends the image 20013 to the computer 2003 through the wireless communication module.

[0410] After the smart TV 2002 receives the image 20012, it obtains the application identifier corresponding to the image 20012, and finds the single-fold enhanced super-resolution model for Game A based on the application identifier of Game A. Then, it selects the single-fold enhanced super-resolution model for Game A as the target super-resolution model.

[0411] The NPU of the smart TV 2002 inputs the image 20012 into the target super-resolution model frame by frame, and performs super-resolution reconstruction on the image 20012 through the target super-resolution model to obtain the image 20021. The image resolution of the image 20021 is 1920×1080.

[0412] As Figure 21 shown, the image resolutions of both the image 20012 and the image 20021 are 1920×1080, but the clarity of the image 20021 is higher than that of the image 20012, and the image quality of the image 20021 is higher than that of the image 20012.

[0413] After the smart TV 2002 obtains the image 20012, it transmits the image 20012 frame by frame to the display screen of the smart TV 2002 for on-screen display.

[0414] After the computer 2003 receives the image 20013, it obtains the application identifier corresponding to the image 20013, and finds the single-fold enhanced super-resolution model for Game A based on the application identifier of Game A. Then, it selects the single-fold enhanced super-resolution model for Game A as the target super-resolution model.

[0415] The NPU of the computer 2003 inputs the image 20013 into the target super-resolution model frame by frame, and performs super-resolution reconstruction on the image 20013 through the target super-resolution model to obtain the image 20031. The image resolution of the image 20031 is 2560×1440.

[0416] As Figure 21 shown, the image resolutions of both the image 20013 and the image 20031 are 2560×1440, but the clarity of the image 20031 is higher than that of the image 20013, and the image quality of the image 20031 is higher than that of the image 20013.

[0417] After the computer 2003 obtains the image 20031, it transmits the image 20031 frame by frame to the display screen of the computer 2003 for on-screen display.

[0418] Scenario Eight:

[0419] Please refer to Figure 22 and Figure 23 , the user operates the mobile phone 2201 to cast the display screen of the mobile phone 2201 to the smart TV 2202 and the computer 2203.

[0420] When the user clicks on the icon of Game A on the mobile phone 2201, the mobile phone 2201 responds to the user's click operation and launches Game A.

[0421] After Game A is launched, the application program of Game A sends the native image data 22011 frame by frame to the GPU of the mobile phone 2201. The image resolution of the native image data 22011 is 480×360.

[0422] The mobile phone 2201 renders the above native image data 22011 frame by frame according to the pre-configured image resolution to obtain the image 22012, and the resolution of the image 22012 is 480×360.

[0423] The mobile phone 2201 sends the image 22012 to the smart TV 2202 through the wireless communication module, and sends the image 22012 to the computer 2203 through the wireless communication module.

[0424] After receiving the image 22012, the smart TV 2202 obtains the application identifier corresponding to the image 22012, finds the single-fold enhanced super-resolution model for Game A according to the application identifier of Game A, and then selects the single-fold enhanced super-resolution model for Game A as the target super-resolution model.

[0425] The NPU of the smart TV 2202 upsamples the image 22012 frame by frame to obtain the image 22021, and the image resolution of the image 22021 is 1920×1080. Then, the NUP of the smart TV 2202 inputs the image 22021 into the target super-resolution model frame by frame, and performs super-resolution reconstruction on the image 22021 through the target super-resolution model to obtain the image 22022.

[0426] As Figure 23 shown, the image resolutions of both the image 22021 and the image 22022 are 1920×1080, but the clarity of the image 22022 is higher than that of the image 22021, and the picture quality of the image 22022 is higher than that of the image 22021.

[0427] After obtaining the image 22022, the smart TV 2202 transfers the image 22022 frame by frame to the display screen of the smart TV 2202 for on-screen display.

[0428] After receiving the image 22012, the computer 2203 obtains the application identifier corresponding to the image 22012, finds the single-fold enhanced super-resolution model for Game A according to the application identifier of Game A, and then selects the single-fold enhanced super-resolution model for Game A as the target super-resolution model.

[0429] The NPU of computer 2203 upsamples the image 22012 frame by frame to obtain the image 22031, and the image resolution of the image 22031 is 2560×1440. Then, the NUP of computer 2203 inputs the image 22031 into the target super-resolution model frame by frame, and performs super-resolution reconstruction on the image 22031 through the target super-resolution model to obtain the image 22032.

[0430] As Figure 23 shown, the image resolutions of both the image 22031 and the image 22032 are 1920×1080, but the clarity of the image 22032 is higher than that of the image 22031, and the image quality of the image 22032 is higher than that of the image 22031.

[0431] After computer 2203 obtains the image 22032, it transmits the image 22032 frame by frame to the display screen of computer 2203 for on-screen display.

[0432] Scene Nine:

[0433] Please refer to Figure 24 and Figure 25 . The user operates the mobile phone 2401 to project the display screen of the mobile phone 2401 to the smart TV 2402 and the computer 2403.

[0434] The user clicks on the icon of Game A on the mobile phone 2401, and the mobile phone 2401 responds to the user's click operation and launches Game A.

[0435] After Game A is launched, the application program of Game A sends the native image data 24011 to the GPU of the mobile phone 2401 frame by frame.

[0436] The mobile phone 2401 renders the above native image data 24011 frame by frame according to the pre-configured image resolution to obtain the image 24012, and the resolution of the image 24012 is 480×360.

[0437] The mobile phone 2401 sends the image 24012 to the smart TV 2402 through the wireless communication module, and sends the image 24012 to the computer 2403 through the wireless communication module.

[0438] After the smart TV 2402 receives the image 24012, it obtains the application identifier corresponding to the image 24012, and finds the multi-fold enhanced super-resolution model for Game A according to the application identifier of Game A, and then selects the multi-fold enhanced super-resolution model for Game A as the target super-resolution model.

[0439] The NPU of the smart TV 2402 inputs the image 24012 into the target super-resolution model frame by frame, and performs super-resolution reconstruction on the image 24012 through the target super-resolution model to obtain the image 24021. The image resolution of the image 24021 is 1920×1080.

[0440] As Figure 25 shown, the image resolution of the image 24021 is greater than that of the image 24012, and the clarity of the image 24021 is greater than that of the image 24012. Therefore, the picture quality of the image 24021 is higher than that of the image 24012.

[0441] After the smart TV 2402 obtains the image 24021, it transmits the image 24021 frame by frame to the display screen of the smart TV 2402 for on-screen display.

[0442] After the computer 2403 receives the image 24012, it obtains the application identifier corresponding to the image 24012, and finds the multi-fold enhanced super-resolution model for Game A according to the application identifier of Game A. Then, it selects the multi-fold enhanced super-resolution model for Game A as the target super-resolution model.

[0443] The NPU of the computer 2403 inputs the image 24012 into the target super-resolution model frame by frame, and performs super-resolution reconstruction on the image 24012 through the target super-resolution model to obtain the image 24031. The image resolution of the image 24031 is 2560×1440.

[0444] The image resolution of the image 24031 is greater than that of the image 24012, and the clarity of the image 24031 is greater than that of the image 24012. Therefore, the picture quality of the image 24031 is higher than that of the image 24012.

[0445] After the computer 2403 obtains the image 24031, it transmits the image 24031 frame by frame to the display screen of the computer 2403 for on-screen display.

[0446] In summary, in the image processing method provided in this embodiment, the first electronic device first performs preliminary rendering on the native image data of the application program to obtain a first image. Then, the first electronic device transmits the first image to the designated device that needs screen mirroring. The designated device performs super-resolution reconstruction on the first image to improve the image quality of the first image and obtain a target image for on-screen display. That is to say, in the image processing method provided in this embodiment, the image rendering process is divided into two steps: preliminary rendering and super-resolution reconstruction. The preliminary rendering step is executed by the first electronic device of the screen mirroring party, and the super-resolution reconstruction step is executed by the designated device of the receiving party of the screen mirroring, thereby reducing the load on the hardware resources of the first electronic device of the screen mirroring party, reducing the rendering power consumption, and making full use of the hardware resources of the first electronic device of the screen mirroring party and the hardware resources of the designated device of the receiving party of the screen mirroring.

[0447] In addition, by using the image processing method provided in this embodiment, the display resolution of the target image after image quality enhancement can be adapted to the display resolution of the designated device, improving the user experience.

[0448] Among them, when the designated device performs super-resolution reconstruction, the super-resolution model selected by the designated device can be a single-fold enhanced super-resolution model, or it can also be a multi-fold enhanced super-resolution model. When the super-resolution model selected by the designated device is a multi-fold enhanced super-resolution model, the hardware resources occupied by the preliminary rendering and the rendering power consumption of the preliminary rendering can be reduced.

[0449] In addition, the above super-resolution model can be a general super-resolution model, or it can also be a specific super-resolution model for a certain application program or a certain type of application program. The applicable range of the specific super-resolution model is smaller than that of the general super-resolution model, but usually the image quality enhancement effect of the specific super-resolution model is better than that of the general super-resolution model.

[0450] It should be understood that the sequence numbers of the steps in the above embodiments do not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

[0451] Please refer to Figure 26 , this embodiment of the present application also provides an electronic device. As Figure 26 shown, the electronic device 26 in this embodiment includes: a processor 260, a memory 261, and a computer program 262 stored in the memory 261 and executable on the processor 260. When the processor 260 executes the computer program 262, it implements the steps in the above embodiment of the screen expansion method, such as Figure 1 the steps S301 to S302 shown. Or, when the processor 260 executes the computer program 262, it implements the functions of each module / unit in the above device embodiments, such asFigure 26 The functions of the modules 2601 to 2602 shown.

[0452] Exemplarily, the computer program 262 can be divided into one or more modules / units, which are stored in the memory 261 and executed by the processor 260 to complete this application. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, and these instruction segments are used to describe the execution process of the computer program 262 in the electronic device 26. For example, the computer program 262 can be divided into a native data module, a preliminary rendering module, and a first super-resolution module. The specific functions of each module are as follows:

[0453] The native data module is used to obtain native image data, where the native image data is image data generated by an application and not yet rendered.

[0454] The preliminary rendering module is used to render the native image data through a first graphics rendering hardware to obtain a first image.

[0455] The first super-resolution module is used to perform super-resolution reconstruction on the first image through a second graphics rendering hardware to obtain a target image, where the first graphics rendering hardware and the second graphics rendering hardware are different graphics rendering hardwares.

[0456] The electronic device 26 can be a computing device such as a desktop computer, a notebook, a handheld computer, and a cloud server. The electronic device may include, but is not limited to, a processor 260 and a memory 261. Those skilled in the art can understand that Figure 26 merely examples of the electronic device 26, which do not constitute a limitation on the electronic device 26, and may include more or fewer components than shown in the figure, or combine certain components, or different components. For example, the electronic device may further include input / output devices, network access devices, buses, etc.

[0457] The so-called processor 260 may be a central processing unit (CPU), or may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0458] The memory 261 may be an internal storage unit of the electronic device 26, such as a hard disk or memory of the electronic device 26. The memory 261 may also be an external storage device of the electronic device 26, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. equipped on the electronic device 26. Further, the memory 261 may also include both an internal storage unit and an external storage device of the electronic device 26. The memory 261 is used to store the computer program and other programs and data required by the electronic device. The memory 261 may also be used to temporarily store data that has been output or is to be output.

[0459] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the above division of each functional unit and module is used as an example. In actual applications, the above functions can be allocated to different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit. In addition, the specific names of each functional unit and module are only for the convenience of mutual distinction and do not limit the protection scope of the present application. The specific working processes of the units and modules in the above system can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0460] In the above embodiments, the descriptions of the various embodiments have their own emphases. For the parts not detailed or recorded in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0461] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or by a combination of computer software and electronic hardware. Whether these functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0462] In the embodiments provided in the present application, it should be understood that the disclosed device / electronic device and method can be implemented in other ways. For example, the device / electronic device embodiments described above are merely illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or hardware can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections between each other can be through some interfaces. The indirect couplings or communication connections of the devices or units can be in electrical, mechanical or other forms.

[0463] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0464] In addition, in each embodiment of the present application, the functional units can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated units can be implemented in the form of hardware or in the form of software functional units.

[0465] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, to implement all or part of the processes in the above method embodiments of the present application, it can also be completed by a computer program instructing the relevant hardware. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content included in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0466] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. An image processing method, characterized in that, applied to a first electronic device, including: The first electronic device acquires original image data, where the original image data is image data generated by an application and not rendered; The first electronic device renders the original image data through a first graphics rendering hardware to obtain a first image; The first electronic device performs super-resolution reconstruction on the first image through a second graphics rendering hardware to obtain a target image, where the first graphics rendering hardware and the second graphics rendering hardware are different graphics rendering hardware; Among them, the first electronic device performs super-resolution reconstruction on the first image through the first graphics rendering hardware to obtain a target image, including: The first electronic device acquires an identifier of the application; The first electronic device searches for a target super-resolution model associated with the identifier; The first electronic device performs super-resolution reconstruction on the first image through the first graphics rendering hardware and the found target super-resolution model to obtain a target image.

2. The image processing method according to claim 1, characterized in that, after the first electronic device searches for a target super-resolution model associated with the identifier, it further includes: if no target super-resolution model associated with the identifier is found, the first electronic device performs super-resolution reconstruction on the first image through the first graphics rendering hardware and a preset general super-resolution model to obtain a target image.

3. The image processing method according to claim 1, characterized in that, the first electronic device renders the original image data through the first graphics rendering hardware to obtain a first image, including: The first electronic device renders the original image data through the first graphics rendering hardware and a preset first image resolution to obtain a first image.

4. The image processing method according to claim 3, characterized in that, the first electronic device performs super-resolution reconstruction on the first image through the second graphics rendering hardware to obtain a target image, including: The first electronic device performs super-resolution reconstruction on the first image through the second graphics rendering hardware and a single-fold enhanced super-resolution model to obtain a target image, where the first image resolution is the same as the image resolution of the target image, and the single-fold enhanced super-resolution model is a super-resolution model in which the image resolution of the input image is the same as the image resolution of the output image.

5. The image processing method according to claim 3, characterized in that, the first electronic device performs super-resolution reconstruction on the first image through the second graphics rendering hardware to obtain a target image, including: The first electronic device performs super-resolution reconstruction on the first image through the second graphics rendering hardware and a multi-fold enhanced super-resolution model to obtain a target image, where the first image resolution is less than the image resolution of the target image, and the multi-fold enhanced super-resolution model is a super-resolution model in which the image resolution of the input image is less than the image resolution of the output image.

6. The image processing method according to claim 1, characterized in that, The first electronic device renders the native image data through a first graphics rendering hardware to obtain a first image, including: The first electronic device renders the native image data through a graphics processing unit to obtain a first image; Correspondingly, the first electronic device performs super-resolution reconstruction on the first image through a second graphics rendering hardware to obtain a target image, including: The first electronic device performs super-resolution reconstruction on the first image through a neural network processor to obtain a target image.

7. An image processing method, characterized in that, applied to a second electronic device, including: The second electronic device receives a first image sent by the first electronic device, where the first image is an image obtained after the first electronic device renders native image data generated by an application; The second electronic device performs super-resolution reconstruction on the first image to obtain a target image; Wherein, the second electronic device performs super-resolution reconstruction on the first image to obtain a target image, including: The second electronic device obtains an identifier of the application; The second electronic device searches for a target super-resolution model associated with the identifier; The second electronic device performs super-resolution reconstruction on the first image through the found target super-resolution model to obtain a target image.

8. The image processing method according to claim 7, characterized in that, after the second electronic device searches for a target super-resolution model associated with the identifier, it further includes: If no target super-resolution model associated with the identifier is found, the second electronic device performs super-resolution reconstruction on the first image through a preset general super-resolution model to obtain a target image.

9. The image processing method according to claim 7, characterized in that, The first resolution of the first image is the same as the image resolution of the target image; The second electronic device performs super-resolution reconstruction on the first image to obtain a target image, including: The second electronic device performs super-resolution reconstruction on the first image through a single-fold enhanced super-resolution model to obtain the target image, where the single-fold enhanced super-resolution model is a super-resolution model in which the image resolution of the input image is the same as the image resolution of the output image.

10. The image processing method according to claim 7, characterized in that, The first resolution of the first image is lower than the image resolution of the target image; The second electronic device performs super-resolution reconstruction on the first image to obtain a target image, including: The second electronic device performs upsampling processing on the first image to obtain a second image, where the image resolution of the second image is the same as the image resolution of the target image; The second electronic device performs super-resolution reconstruction on the second image through a single-fold enhanced super-resolution model to obtain the target image, where the single-fold enhanced super-resolution model is a super-resolution model in which the image resolution of the input image is the same as the image resolution of the output image.

11. The image processing method according to claim 7, characterized in that, The first resolution of the first image is lower than the resolution of the target image; The second electronic device performs super-resolution reconstruction on the first image to obtain a target image, including: The second electronic device performs super-resolution reconstruction on the first image through a multi-fold enhanced super-resolution model to obtain the target image, where the multi-fold enhanced super-resolution model is a super-resolution model in which the image resolution of the input image is smaller than the image resolution of the output image.

12. An image processing system, characterized in that the system includes a first electronic device and a second electronic device; The first electronic device is configured to render the native image data generated by the application program to obtain a first image, and send the first image to the second electronic device; The second electronic device is configured to execute the image processing method according to any one of claims 7 to 11.

13. An electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that when the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented, or the method according to any one of claims 7 to 11 is implemented.

14. A computer-readable storage medium storing a computer program, characterized in that when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented, or the method according to any one of claims 7 to 11 is implemented.

15. A chip system, characterized in that the chip system includes a memory and a processor, and the processor executes the computer program stored in the memory to implement the method according to any one of claims 1 to 6, or to implement the method according to any one of claims 7 to 11.

Images