Image processing method and electronic device

Abstract

An embodiment of the present application provides an image processing method and electronic device, which is performed by the electronic device and includes: obtaining an image to be displayed on the electronic device, the resolution of the image to be displayed being a first resolution; performing resolution reduction processing on the image to be displayed to obtain a first image, the resolution of the first image being a second resolution, the first resolution being greater than the second resolution; inputting the first image into a neural network model for image super-resolution processing to obtain a second image, and displaying the second image, the resolution of the second image being the first resolution, and the picture quality of the second image being better than the picture quality of the image to be displayed. In this method, after the image super-resolution processing performed by the neural network model, the image can be optimized to improve key information such as image quality, details, and edges of objects and characters, thereby improving the picture quality in, for example, gaming scenes.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method and electronic device. Background Art

[0002] Currently, wireless electronic devices are becoming increasingly versatile. For example, mobile devices such as mobile phones and tablets can not only meet users' daily communication and office needs, but can also run real-time online games to improve the user experience.

[0003] However, for online games that require a high level of real-time performance, they place high demands on key information such as image quality, details, and edges between objects and characters. Poor image quality can seriously impact the user's gaming experience. Therefore, research is needed to improve the image quality of mobile electronic devices in gaming scenarios. Summary of the Invention

[0004] The present application provides an image processing method and electronic device, which can significantly improve the picture quality of the electronic device in a gaming scenario.

[0005] In a first aspect, the present application provides an image processing method, which is executed by an electronic device, and includes: obtaining an image to be displayed of the electronic device, the resolution of the image to be displayed being a first resolution; reducing the resolution of the image to be displayed to obtain a first image, the resolution of the first image being a second resolution, and the first resolution being greater than the second resolution; inputting the first image into a neural network model for image super-resolution processing to obtain a second image, and displaying the second image, the resolution of the second image being the first resolution, and the picture quality of the second image being better than the picture quality of the image to be displayed.

[0006] The image to be displayed on the electronic device is the image to be displayed on the display screen of the electronic device. For example, when an application is running on the electronic device, the image to be displayed is the display screen of the application. For example, the image to be displayed can be the display screen of a game application running on the electronic device.

[0007] After acquiring the image to be displayed, the electronic device can first reduce the resolution of the image to be displayed, for example, it can use a downsampling method to reduce the resolution to obtain a first image. Afterwards, the electronic device can use the trained neural network model to perform image super-resolution processing on the first image to obtain a second image with the same resolution as the image to be displayed. Here, the neural network model used in this application can realize the conversion process from low-resolution images to high-resolution images, and the neural network model focuses on processing the details of the objects contained in the image during training. For example, the features of the objects or characters contained in the image can be refined to make the picture quality of the second image better.

[0008] In the above implementation, the electronic device can first reduce the resolution of the image to be displayed, then use the trained neural network model to perform image super-resolution processing on this low-resolution image to obtain a high-resolution image with the same resolution as the original image to be displayed, and then display this high-resolution image. After the image super-resolution processing performed by the neural network model, the image can be optimized to improve key information such as image quality, details, and edges of objects and characters, thereby improving the image quality in gaming scenarios.

[0009] In combination with the first aspect, in some implementations of the first aspect, the above-mentioned inputting the first image into the neural network model for image super-resolution processing to obtain the second image includes: performing an interpolation and amplification operation on the first image to obtain an enlarged first image, and the size of the enlarged first image is the same as the size of the second image; inputting the enlarged first image into the neural network model for image super-resolution processing to obtain the second image.

[0010] The electronic device may firstly enlarge the first image to the same size as the second image, and then perform image super-resolution processing on the enlarged first image to obtain the second image.

[0011] In this implementation, to reduce problems such as insufficient memory caused by storing the first image after amplification, the electronic device can perform image rearrangement (space to depth) on the amplified first image after obtaining it, obtaining multiple small images arranged according to image depth; the electronic device can then input the multiple small images into a neural network model for image super-resolution processing, obtaining output results corresponding to the multiple small images, for example, the output results are multiple vector matrices; if a result corresponding to the amplified first image is to be obtained, it is necessary to perform the inverse operation of the above-mentioned image rearrangement (depth to space) on the output results corresponding to the multiple small images, that is, to obtain a high-resolution second image, the size of which is the same as that of the amplified first image. In this way, the electronic device can perform reasonable memory allocation and reduce the occurrence of memory problems.

[0012] In combination with the first aspect, in some implementations of the first aspect, before inputting the first image into the neural network model for image super-resolution processing, the above method also includes: obtaining a training image set, the training image set including an input image and a supervisory image, the resolution of the supervisory image being greater than the resolution of the input image; inputting the input image into the neural network model to be trained to obtain an output image; calculating the loss between the output image and the supervisory image, and adjusting the network parameters of the neural network model to be trained according to the loss to train the neural network model.

[0013] From the above description, it can be seen that the neural network model used by the electronic device is a trained network model. Therefore, before using the neural network model, it needs to be trained to improve the accuracy of the processing results.

[0014] In this implementation, the electronic device can obtain a large number of training image sets, wherein the input images are used to input into the neural network model for processing, and the supervisory images are used to monitor the difference between the output of the neural network model and the supervisory images, thereby supervising the learning and training process of the neural network model. In one implementation, the resolution of the supervisory images is greater than that of the input images, that is, the input images include low-resolution images, and the supervisory images include high-resolution images. The high-resolution images are used as supervisory signals, and the difference between the output images of the neural network model and the high-resolution images is calculated to train the neural network model.

[0015] Optionally, the input image may include multiple images with different resolutions, so that the neural network model can be trained for images with different resolutions to adapt to inputs with different resolutions and improve model compatibility.

[0016] During model training, the loss between the output image of the neural network model and the supervised image is typically calculated, and the network parameters of the neural network model are continuously adjusted based on the loss. Loss functions are typically varied. In this application, in conjunction with the first aspect, in some implementations of the first aspect, the loss includes at least one of a first loss, a second loss, a third loss, and a fourth loss.

[0017] The first loss represents the error between the multiple vector matrices corresponding to the output image and the multiple vector matrices corresponding to the supervision image. The second loss represents the error between the features of the output image and the features of the supervision image. The third loss represents the error between the edge features of the objects contained in the output image and the edge features of the objects contained in the supervision image. The fourth loss represents the error between the pixel values ​​of the pixels in the output image and the pixel values ​​of the pixels in the supervision image.

[0018] Among them, the electronic device can use one or more loss functions to calculate the loss between the output image and the supervision image to supervise the edges, details, and multi-dimensionality of objects such as characters, ensuring that the game screen can be improved in game scenes such as character edges and distant views.

[0019] The first loss can be the regression loss function value of the output result. Optionally, the electronic device can calculate the regression loss function value according to Regression Loss = tf.generator.losses(). The second loss can be the output feature loss function value. Optionally, the electronic device can input the output image and the supervision image into the feature extraction network for feature extraction, and calculate the loss based on the extracted features. The third loss can be the edge loss function value. Optionally, it can be the edge loss function value based on the Sobel operator. The fourth loss can be the pixel-level loss function value, that is, for the output image and the supervision image, the pixel value error of the pixel points at the corresponding position in the image.

[0020] In this implementation, the electronic device calculates the loss between the output image and the supervision image by using one or more loss functions to supervise the edges, details, and multi-layer dimensions of people, etc., thereby improving the processing accuracy of the neural network model and enabling higher-quality images to be obtained during subsequent image super-resolution processing.

[0021] In combination with the first aspect, in some implementations of the first aspect, after obtaining the training image set, the above method also includes: performing image super-resolution processing on the supervision image to obtain a supervision image with improved resolution; and performing resolution reduction processing on the input image to obtain an input image with reduced resolution.

[0022] In order to improve the neural network model's ability to handle more image quality problems and improve the accuracy of the neural network model's output results, electronic devices can perform data enhancement preprocessing on the training image set, such as reducing the clarity of the input image (low-resolution image) to make it have more image quality problems, and improving the clarity of the supervision image (high-resolution image) to ensure that the neural network model can ultimately output more accurate results.

[0023] Therefore, electronic devices can perform image super-resolution processing on the supervision image to improve the resolution of the supervision image, so that the edges of people and objects in the obtained supervision image are smoother and clearer. It is also possible to reduce the resolution of the input image, such as downsampling, to obtain an input image with a lower resolution.

[0024] In combination with the first aspect, in some implementations of the first aspect, before reducing the resolution of the image to be displayed, the above method also includes: performing text recognition on the image to be displayed; if it is recognized that the image to be displayed contains text, the text is separated from the image to be displayed to obtain the separated text and the image to be displayed that only contains the image.

[0025] Because in some scenarios (such as game scenarios), the image to be displayed may contain text, if the image to be displayed with text is subjected to reduced resolution or super-resolution processing, it may cause text distortion and other phenomena. Therefore, before processing the image to be displayed, the present application can also separate the text. Optionally, the electronic device can use a text recognition algorithm to identify the image to be displayed to determine whether the image to be displayed contains text; when the electronic device recognizes that the image to be displayed contains text, it can separate the text, for example, using a segmentation algorithm to segment the text, to obtain the separated text and the image to be displayed that only contains the image.

[0026] Correspondingly, in the case of separating text, after inputting the first image into the neural network model for image super-resolution processing to obtain the second image, the above method also includes: splicing the second image and the separated text to obtain a second image containing text.

[0027] In this implementation, since the text in the image to be displayed does not participate in the image processing process, the displayed text is still the original text, which does not affect the display effect of the text and further improves the picture quality of the final displayed image.

[0028] In combination with the first aspect, in some implementations of the first aspect, the electronic device includes an embedded neural network processing unit (NPU), and the step of inputting the first image into the neural network model for image super-resolution processing to obtain the second image includes: inputting the first image into the neural network model through the NPU for image super-resolution processing to obtain the second image.

[0029] Among them, since the traditional image rendering process is performed on the graphics processing unit (GPU), this undoubtedly increases the load on the GPU. In this regard, the present application transfers the image super-resolution processing process performed by the neural network model to the NPU for processing. The GPU can only process the process of reducing the resolution of the image to be displayed and rendering it, thereby reducing the load on the GPU, that is, reducing the processing power consumption of the electronic device.

[0030] In addition, the electronic device is also built with GTX (GPU turbo X) framework and The QNN (qualcomm neural network) framework builds an AP library (libsr.so) and a CP library (libsr_skel.so) in the QNN framework. The relevant algorithms of the neural network model for image super-resolution processing can be integrated in the AP library and CP library. The operations of the CP library can be transferred to the NPU for processing and for the GTX framework to call.

[0031] In a second aspect, the present application provides a device, which is included in an electronic device and has the function of implementing the electronic device behavior described in the first aspect and possible implementations of the first aspect. The function can be implemented through hardware or through hardware executing corresponding software implementations. The hardware or software includes one or more modules or units corresponding to the above functions. For example, a receiving module or unit, a processing module or unit, etc.

[0032] In a third aspect, the present application provides an electronic device, which includes: a processor, a memory, and an interface; the processor, the memory, and the interface cooperate with each other so that the electronic device executes any one of the methods in the technical solution of the first aspect.

[0033] In a fourth aspect, the present application provides a chip including a processor, wherein the processor is configured to read and execute a computer program stored in a memory to invoke instructions so that an electronic device executes the method in the first aspect and any possible implementation thereof.

[0034] Optionally, the chip further includes a memory, and the memory is connected to the processor via a circuit or wire.

[0035] Further optionally, the chip also includes a communication interface.

[0036] In a fifth aspect, the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the processor calls instructions to enable the electronic device to execute any one of the methods in the technical solution of the first aspect.

[0037] In a sixth aspect, the present application provides a computer program product, which includes: a computer program code, which, when the computer program code runs on an electronic device, enables the electronic device to execute any one of the methods in the technical solution of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a schematic diagram of resource allocation of traditional technology and this application;

[0039] Figure 2 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0040] Figure 3 This is a software structure diagram of an electronic device provided in an embodiment of the present application;

[0041] Figure 4 This is a flowchart of an image processing method provided in an embodiment of the present application;

[0042] Figure 5 This is a schematic diagram of an image processing method provided in an embodiment of the present application;

[0043] Figure 6 is a flowchart of another image processing method provided in an embodiment of the present application;

[0044] Figure 7 This is a schematic diagram of the training process of a neural network model provided in an embodiment of the present application;

[0045] Figure 8 This is a flowchart of another image processing method provided in an embodiment of the present application;

[0046] Figure 9 This is a comparison chart of the display effects of the game screen when running in different scenarios provided by the embodiment of the present application. DETAILED DESCRIPTION

[0047] The technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings in the embodiments of the present application. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships can exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0048] In the following, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the quantity of the technical features indicated. Therefore, a feature specified as "first," "second," or "third" may explicitly or implicitly include one or more of the features.

[0049] Traditional online games are usually run on electronic devices with strong computing power, such as personal computers (PCs) and ultra-mobile personal computers (UMPCs). However, with the continuous development of electronic technology, the performance of mobile electronic devices has been greatly improved. Online games with high real-time requirements can also be run on mobile electronic devices, such as mobile phones and tablet computers.

[0050] For scenarios where online games are played on mobile electronic devices, an embodiment of the present application provides an image processing method that can first reduce the resolution of an image to be displayed obtained from a game application, and then use a trained neural network (NN) model to perform image superresolution processing (also referred to as super-resolution processing) on ​​the low-resolution image to obtain a high-resolution image with the same resolution as the original image to be displayed, and then display the high-resolution image. After the image processing method of the embodiment of the present application, the image displayed by the game application is an image that has been super-resolution processed. The super-resolution processing performed by the neural network can optimize the image, improve key information such as image quality, details, and edges of objects and characters, thereby improving the picture quality in the game scene.

[0051] As we all know, there are currently several image super-resolution algorithms. For example, one uses mathematical calculations using non-AI methods, including FSR (AMD fidelityFX super resolution), MSR (meta super resolution), and algorithms based on ray tracing. Another method uses AI deep neural networks for pixel prediction, such as DLSS (Nvidia deep learning super sampling). Among them, although FSR and MSR algorithms can be used by mobile electronic devices, they mainly predict and calculate high-resolution images by analyzing adjacent pixels in the image, which may have certain image quality issues. Especially in gaming scenarios that require extra attention to detail and image quality, FSR and MSR cannot meet the image quality requirements of current games. Ray Tracing-based algorithms and DLSS have a large amount of computation and power consumption during the calculation process, and cannot be directly applied to mobile electronic devices.

[0052] The neural network model provided in the embodiment of the present application is a lightweight network that can be well oriented to gaming scenarios and can be applied to mobile electronic devices. It should be noted that the image processing method provided in the embodiment of the present application can be applied to mobile electronic devices with the above-mentioned super-resolution processing capabilities, such as mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, laptops, netbooks, personal digital assistants (PDAs), etc. The embodiment of the present application does not impose any restrictions on the specific type of electronic devices.

[0053] Furthermore, if Figure 1 As shown, in traditional online game scenarios, the GPU is usually used to render and display the game screen, which undoubtedly increases the GPU load. In response to this, the embodiments of the present application also propose to transfer the image super-resolution processing performed by the neural network model to the NPU for processing. The GPU can only handle the process of reducing the resolution of the image to be displayed and rendering it, thereby reducing the GPU load, that is, reducing the processing power consumption of the electronic device.

[0054] For example, Figure 21 is a schematic diagram of the structure of an electronic device 100 provided in an embodiment of the present application. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display 194, and a subscriber identification module (SIM) card interface 195. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0055] The processor 110 may include one or more processing units, for example, an application processor (AP), a modem processor, a GPU, an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or an NPU. The different processing units may be independent devices or integrated into one or more processors.

[0056] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0057] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly retrieve it from the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0058] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface.

[0059] It is understood that the interface connection relationship between the modules illustrated in the embodiments of the present application is merely an illustrative illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods from the above embodiments, or a combination of multiple interface connection methods.

[0060] The charging management module 140 is configured to receive charging input from a charger. The charger can be either a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 can receive charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 can receive wireless charging input via the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also provide power to the electronic device via the power management module 141.

[0061] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and provides power to the processor 110, the internal memory 121, the external memory, the display 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage, impedance). In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0062] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0063] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Figure 2 The structures of antenna 1 and antenna 2 are merely examples. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In other embodiments, the antennas can be used in conjunction with tuning switches.

[0064] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0065] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, frequency modulate it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0066] Electronic device 100 implements display functionality through a GPU, display screen 194, and an application processor. A GPU is a microprocessor for image processing that connects display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0067] Display screen 194 is used to display images, videos, and the like. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-oLed, or a quantum dot light-emitting diode (QLED). In some embodiments, electronic device 100 may include one or N display screens 194, where N is a positive integer greater than one.

[0068] The electronic device 100 can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.

[0069] The NPU is a neural network computing processor. Drawing on the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it rapidly processes input information and can continuously self-learn. The NPU can enable intelligent cognitive applications in electronic device 100, such as image recognition, face recognition, voice recognition, and text comprehension.

[0070] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 via the external memory interface 120 to implement data storage functions. For example, files such as music and videos can be stored on the external memory card.

[0071] The internal memory 121 can be used to store computer executable program codes, and the executable program codes include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area can store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0072] The electronic device 100 can implement audio functions such as music playback and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0073] The buttons 190 include a power button, a volume button, and the like. The buttons 190 may be mechanical buttons or touch buttons. The electronic device 100 may receive key inputs and generate key signal inputs related to user settings and function control of the electronic device 100.

[0074] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 194, motor 191 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0075] The indicator 192 may be an indicator light, which may be used to indicate the charging status, power level changes, messages, missed calls, notifications, etc.

[0076] It should be understood that the structures illustrated in the embodiments of the present application do not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0077] The software system of the electronic device 100 can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. In the embodiment of the present application, the Android system with a layered architecture is used as an example to illustrate the software structure of the electronic device 100.

[0078] Figure 3 This is a block diagram of the software structure of the electronic device 100 according to an embodiment of the present application. A layered architecture divides software into several layers, each with distinct roles and responsibilities. Layers communicate with each other via software interfaces. In some embodiments, the Android system is divided into four layers: the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer.

[0079] like Figure 3 As shown, the application layer can include a series of application packages. Application packages can include games, cameras, gallery, calendar, calls, maps, navigation, WLAN, Bluetooth, music, video, short messages and other applications. For the convenience of illustration, Figure 3 Only game applications are used as an example.

[0080] The application framework layer provides an application programming interface (API) and programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0081] like Figure 3As shown, the application framework layer can build a GTX (GPU turbo X) framework, which is generally capable of performing game logic processing and global optimization rendering processing. In one embodiment, the GTX framework may include a data capture module, a scene analysis module, a model determination module, an operator optimization module, a model operation module, and an image output module.

[0082] Among them, the data capture module can capture the data of the image to be displayed from the game application, and can reduce the resolution of the image to be displayed. The scene analysis module can identify the image to be displayed and perform text separation and other processing on the image containing text. The model determination module can determine the input resolution and output resolution of the neural network model based on the resolution of the image after resolution reduction, the display resolution of the electronic device or the display resolution of the game application, so that the subsequent model operation module can perform operations based on the input resolution and output resolution. The operator optimization module can perform memory operator optimization before inputting the image to be displayed into the neural network model, such as rearranging the image to be displayed to reduce memory usage. The model operation module can use the neural network model to perform image super-resolution operation on the input low-resolution image to obtain a high-resolution image with the same resolution as the original image to be displayed. The image output module can be used to splice the output high-resolution image and the separated text to obtain the final displayed image.

[0083] In addition, the above application framework layer can also include window manager, content provider, view system, phone manager, resource manager, notification manager, etc. ( Figure 3 Not shown yet).

[0084] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, take screenshots, etc. The content provider is used to store and obtain data and make this data accessible to applications. The data can include video, images, audio, calls made and received, browsing history and bookmarks, phone books, etc. The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. The display interface can be composed of one or more views. For example, the display interface including the SMS notification icon can include a view for displaying text and a view for displaying images. The phone manager is used to provide communication functions for the electronic device 100. For example, call status management (including call connection, hang up, etc.) The resource manager provides various resources to applications, such as localized strings, icons, images, layout files, video files, etc. The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can automatically disappear after a short period of time without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system's top status bar in the form of an icon or scrolling text bar, such as a notification from an application running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message may be displayed in the status bar, a notification sound may be emitted, an electronic device may vibrate, an indicator light may flash, etc.

[0085] The Android runtime includes the core library and the virtual machine. The Android runtime is responsible for scheduling and management of the Android system.

[0086] The core library consists of two parts: one is the function that needs to be called by the Java language, and the other is the Android core library.

[0087] The application layer and application framework layer run in a virtual machine. The virtual machine executes Java files in the application layer and application framework layer as binary files. The virtual machine manages object lifecycles, stack management, thread management, security and exception management, and garbage collection.

[0088] The system library can include multiple functional modules, such as a surface manager, media libraries, a 3D graphics processing library (such as OpenGL ES), and a 2D graphics engine (such as SGL).

[0089] The surface manager is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications.

[0090] The media library supports playback and recording of a variety of common audio and video formats, as well as static image files. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0091] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0092] A 2D graphics engine is a drawing engine for 2D drawings.

[0093] The kernel layer is the layer between hardware and software. It includes at least the display driver, camera driver, and audio driver. For example, the display driver can be used to receive image data transmitted by a gaming application and display the corresponding image.

[0094] Based on the above GTX framework, the embodiment of the present application can realize the process of reducing the resolution of the image to be displayed obtained from the game application, and then using the trained neural network model to perform image super-resolution processing on the low-resolution image to obtain a high-resolution image with the same resolution as the original image to be displayed and then display it. In addition, if the image super-resolution processing performed by the neural network model is to be transferred to the NPU for processing, the embodiment of the present application can also use The QNN framework platform transfers the processing process to reduce the GPU load. The following is a brief introduction to the performance advantages of the QNN framework platform:

[0095] First, compared to other artificial intelligence (AI) frameworks, such as the SNPE (snapdragon neural processing engine) framework, QNN can reduce memory copies, use ion buffers (ION buffers) and register them to the hardware acceleration unit CDSP (compute DSP), which can reduce performance overhead compared to ordinary buffers. For example, experiments have shown that the performance overhead reduced by 1MB is about 0.6mA (milliamperes). Secondly, QNN can increase the usage ratio of CDSP, reduce CPU power consumption compared to GPU, and increase cache utilization. Finally, QNN can also optimize the processing operators HVX (Hexagon vector extension, i.e. Hexagon-hexagon, Vector-vector, Extensions-extension) before and after the neural network model processing through the single instruction multiple data (SIMD) capability of CDSP, thereby improving the operating efficiency of the processing operators.

[0096] Therefore, the embodiment of the present application uses QNN as the basic framework and uses the latest Hexagon SDK version to build an AP library (libsr.so) and a CP library (libsr_skel.so). The AP library is responsible for the external interface, and can perform model parsing and schedule CDSP libraries, etc. (NDK compilation tools can be used here), and the CP library is responsible for the specific super-resolution algorithm implementation, including algorithm logic and related operators, etc. (Hexagon Tool compilation tools can be used here), and the AP library and the CP library can be scheduled through the IDL interface (such as fastRPC). Among them, the relevant algorithms of the neural network model for super-resolution processing of images can be integrated in the AP library and the CP library, and the operations of the CP library can be transferred to the NPU for processing for GTX framework calls.

[0097] Based on the above GTX framework and QNN framework, the image processing method provided by the embodiment of the present application will be described in detail below. For ease of understanding, the following embodiments of the present application will be described with Figure 2 and Figure 3 Taking the electronic device with the structure shown as an example, the image processing method provided in the embodiment of the present application is specifically described in combination with the accompanying drawings and application scenarios. Figure 4 This is a flow chart of an image processing method provided in an embodiment of the present application. The method may include:

[0098] S101: Acquire an image to be displayed on an electronic device.

[0099] The image to be displayed can be a screen to be displayed of a game application currently running on the electronic device. Since the screen of the game application changes in real time, the electronic device can also obtain the image to be displayed at each moment in real time. For example, the image to be displayed at the current moment is obtained, and the resolution of the image is 1080p.

[0100] It should be understood that the image processing method provided in the embodiment of the present application can be applied not only in the running process of game applications, but also in other scenarios, such as video playback scenarios, video conferencing scenarios, and other scenarios with high requirements on the quality of the display images.

[0101] Optionally, the electronic device can capture the image to be displayed of the game application in real time through the GTX framework.

[0102] S102: Reduce the resolution of the image to be displayed to obtain a first image.

[0103] When reducing the resolution of the image to be displayed, the electronic device may employ a downsampling method, such as an MSR (meta super resolution) algorithm, to downsample the image to be displayed to obtain the first image. For example, the electronic device may reduce the resolution of a 1080p image to be displayed to obtain a first image with a 720p resolution.

[0104] It can be understood that since the above process of reducing the resolution of the image to be displayed consumes less power, this step can be retained and executed on the GPU.

[0105] S103: Input the first image into a neural network model for image super-resolution processing to obtain a second image with the same resolution as the image to be displayed, and display the second image.

[0106] The neural network model used in this step is a pre-trained model that can convert low-resolution images to high-resolution images. Furthermore, during training, the neural network model focuses on processing the details of objects contained in the image. For example, it can refine the features of objects or people contained in the image, thereby improving the quality of the resulting high-resolution image. For example, after inputting a first image with a 720p resolution into the neural network model, a second image with a 1080p resolution can be generated, and this second image has better quality than the image to be displayed.

[0107] It should be noted that the above-mentioned neural network model can be applied to image inputs of different resolutions, as long as the set input format is met. For example, the input format can be (H, W, C), where H and W represent the height and width of the input image, respectively, and C represents the number of channels, for example, C = 3. In actual scenarios, because the electronic device has already reduced the resolution of the image to be displayed to the first image, that is, the resolution of the first image is known, the electronic device can pass the resolution of the first image into the neural network model in advance, so that the neural network model can fix the resolution of the input image in this scenario.

[0108] In one embodiment, the neural network model may be a network model including N convolutional layers, for example, a network model including 7 convolutional layers, each of which has corresponding network parameters. Taking a network model with 7 convolutional layers as an example, the network parameters of the neural network model in the embodiment of the present application may be:

[0109] res=tf.keras.layers.Conv2D(filters=F1, kernel_size=K1, strides=S1, padding=P1)(res);

[0110] res=tf.keras.layers.Conv2D(filters=F2, kernel_size=K2, strides=S2, padding=P2)(res);

[0111] res=tf.keras.layers.Conv2D(filters=F3, kernel_size=K3, strides=S3, padding=P3)(res);

[0112] res=tf.keras.layers.Conv2D(filters=F4, kernel_size=K4, strides=S4, padding=P4)(res);

[0113] res=tf.keras.layers.Conv2D(filters=F5, kernel_size=K5, strides=S5, padding=P5)(res);

[0114] res=tf.keras.layers.Conv2D(filters=F6, kernel_size=K6, strides=S6, padding=P6)(res);

[0115] res=tf.keras.layers.Conv2D(filters=F7, kernel_size=K7, strides=S7, padding=P7)(res);

[0116] The above are the network parameters of 7 convolutional layers from top to bottom. filters represents the number of filters in the convolutional layer, that is, the size of the output data of each convolutional layer; kernel_size represents the size of the convolution kernel in the convolutional layer, for example, it can be 3×3; strides represents the size relationship between the input and output data of the convolutional layer. The larger the strides, the smaller the output data is relative to the input data. For example, when strides = 1, the size of the output image and the output image remains unchanged when the input data is an image. When strides is greater than 1, the size of the output image is smaller than the input image. padding represents the data padding method, which usually includes two padding methods: valid and same. The valid method indicates that there is no padding, and the size of the output data may change after convolution. The same method indicates that the output data and the input data are the same size after padding.

[0117] After the electronic device inputs the above-mentioned first image into the neural network model, it can obtain a second image with the same resolution as the image to be displayed through convolution operations of N layers of convolution layers.

[0118] In one embodiment, the process of an electronic device performing image super-resolution processing on a first image using a neural network model may include: the electronic device first performing an interpolation and amplification operation on the first image to obtain an amplified first image, and then inputting the amplified first image into the neural network model for a convolution operation (i.e., image super-resolution processing) to obtain a second image. It should be noted that if the size of the amplified first image is the same as the size of the desired second image, but the resolution of the amplified first image is still smaller than the resolution of the second image, the neural network model is used to increase the resolution of the amplified first image.

[0119] In actual scenarios, due to the memory limitations of electronic devices, if the amplified first image is directly stored and subsequently operated, insufficient memory may occur. Therefore, in order to reasonably allocate memory, in another embodiment, the electronic device can rearrange the amplified first image. Figure 5 As shown, the process of an electronic device performing image super-resolution processing on a first image through a neural network model may include: the electronic device first performs an interpolation and amplification operation on the first image to obtain an amplified first image, and then performs image rearrangement (space to depth) on the amplified first image. It should be noted here that space to depth is to move the spatial data of the image (i.e., width and height dimensions) to the depth dimension, which is equivalent to dividing the amplified first image into multiple small images, and arranging the multiple small images in depth order according to their positions in the first image. Afterwards, the electronic device can input the rearranged image into the neural network model for convolution operation. It can be understood that the input result of the neural network model should be the output result corresponding to the multiple small images, such as multiple vector matrices. If a result corresponding to the amplified first image is to be obtained, it is necessary to perform the inverse operation of the above-mentioned image rearrangement (depth to space) on the output results corresponding to the multiple small images, that is, to obtain a high-resolution second image, the size of which is the same as that of the amplified first image.

[0120] In this embodiment, image reordering can be achieved by using the function res=tf.nn.space_to_depth(inp, M), where M represents the scale of the image reordering. For example, space_to_depth(inp, 3) means reordering the image at 3 scales. In this scenario, the network parameters of the network model with 7 convolutional layers can be:

[0121] res=tf.keras.layers.Conv2D(filters=32, kernel_size=3, strides=1, padding='same')(res);

[0122] res=tf.keras.layers.Conv2D(filters=32, kernel_size=3, strides=1, padding='same')(res);

[0123] res=tf.keras.layers.Conv2D(filters=32, kernel_size=3, strides=1, padding='same')(res);

[0124] res=tf.keras.layers.Conv2D(filters=32, kernel_size=3, strides=1, padding='same')(res);

[0125] res=tf.keras.layers.Conv2D(filters=32, kernel_size=3, strides=1, padding='same')(res);

[0126] res=tf.keras.layers.Conv2D(filters=32, kernel_size=3, strides=1, padding='same')(res);

[0127] res=tf.keras.layers.Conv2D(filters=27, kernel_size=3, strides=1, padding='same')(res);

[0128] In this example, if the image is rearranged at three scales, the number of resulting small images is 27. Therefore, in the final convolutional layer, filters = 27 should be set to ensure that the number of output results corresponding to the final small images is also 27. In this way, when the output results are subjected to the inverse operation of image rearrangement, a second image with the same size as the enlarged first image is obtained. It is understood that the number of filters in the first through sixth convolutional layers is not limited to 32 and can also be other numbers, such as 16 or 64.

[0129] Optionally, in one embodiment, the electronic device may move the above-mentioned image super-resolution processing to the NPU for processing to reduce the GPU load.

[0130] In the aforementioned image processing method, the electronic device can first reduce the resolution of the image to be displayed, then use the trained neural network model to perform image super-resolution processing on this low-resolution image to obtain a high-resolution image with the same resolution as the original image to be displayed, and then display this high-resolution image. After the super-resolution processing performed by the neural network model, the image can be optimized to improve key information such as image quality, details, and edges of objects and characters, thereby enhancing the image quality in gaming scenarios.

[0131] In actual scenarios, for example, when the image to be displayed is a screen to be displayed of a running game application, the image to be displayed usually contains text, such as text prompts in the game, player speeches, etc. If the image to be displayed with text is subjected to resolution reduction or super-resolution processing, it may cause text distortion and other phenomena. Therefore, in the embodiment of the present application, the text can be separated before processing the image to be displayed. Figure 6 FIG. 1 is a flow chart of another image processing method provided in an embodiment of the present application. The method may include:

[0132] S201: Acquire an image to be displayed on an electronic device.

[0133] S202 , performing text recognition on the image to be displayed. If it is recognized that the image to be displayed contains text, the image to be displayed is separated from the text to obtain the separated text and the image to be displayed containing only the image.

[0134] The electronic device may employ a text recognition algorithm to recognize the image to be displayed. The text recognition algorithm may include, but is not limited to, a maximally stable extremal regions (MSER) ​​algorithm, a connectionist text proposal network (CTPN) algorithm, and the like. When the electronic device recognizes that the image to be displayed contains text, it may separate the text, for example, by employing a segmentation algorithm to separate the text, thereby obtaining the separated text and an image to be displayed that contains only the image. Optionally, the segmentation algorithm may include, but is not limited to, an edge segmentation algorithm, a region segmentation algorithm, and the like.

[0135] S203 , performing resolution reduction processing on the image to be displayed that only includes the image to obtain a first image.

[0136] S204: Input the first image into a neural network model for image super-resolution processing to obtain a second image with the same resolution as the image to be displayed.

[0137] The processes from S203 to S204 refer to S102 to S103 in the above embodiment and are not described in detail here.

[0138] S205: Splice the second image and the separated text, and display the spliced ​​image.

[0139] During the text separation process, the electronic device can record the coordinates of the text within the image to be displayed. Once the second image is generated, the text can be spliced ​​onto the second image based on the coordinates of the text, and the resulting spliced ​​image can be displayed. Because the text is not involved in the image processing, the displayed text remains the original text, without affecting the display quality, further improving the quality of the final displayed image.

[0140] For the neural network model used in the above image processing method, before the electronic device performs image super-resolution processing, the neural network model needs to be trained first. After the trained neural network model is converged, it is moved to the electronic device for processing to improve the accuracy of the processing result. Therefore, the training process of the above neural network model will be described in detail in the following embodiments. Figure 7 As shown, the training process may include:

[0141] S301, obtaining a training image set.

[0142] The training image set may include an input image and a supervisory image. The input image is used to input into the neural network model for processing, and the supervisory image is used to monitor the difference between the output of the neural network model and the supervisory image, thereby supervising the learning and training process of the neural network model. It can be understood that one input image can correspond to one supervisory image. Assuming that the resolution of the input image is a and the resolution of the supervisory image is b, then (a, b) is a set of input and output combinations. If the neural network model is subsequently used for actual image super-resolution processing, if a first image with resolution a is input, a second image with resolution b can be obtained.

[0143] It is also understood that the resolutions of the input images in the training image set can be different, and the resolutions of the corresponding supervisory images can also be different; for example, (480p, 1080p) can be a set of input and output combinations, (480p, 720p) can also be a set of input and output combinations, and (720p, 1080p) can also be a set of input and output combinations. In other words, the neural network model of the embodiment of the present application can be applied to inputs of various resolutions, and has greater compatibility.

[0144] In one embodiment, the input image comprises a low-resolution image, and the supervisory image comprises a high-resolution image. The high-resolution image is used as the supervisory signal, and the difference between the output image of the neural network model and the high-resolution image is calculated to train the neural network model. In a gaming application scenario, the training image set acquired by the electronic device can be a large number of images of different game screens. Since the game image quality can usually be set in gaming applications, the electronic device can then acquire low-resolution and high-resolution images by setting different game image qualities.

[0145] In this embodiment, in order to improve the processing capability of the neural network model for more image quality problems and improve the accuracy of the output results of the neural network model, the electronic device can perform data enhancement preprocessing on the training image set, such as reducing the clarity of the input image (low-resolution image) so that it has more image quality problems, and improving the clarity of the supervision image (high-resolution image), to ensure that the neural network model can ultimately output more accurate results.

[0146] As a feasible method, for high-resolution images, electronic devices can use the GTX framework to change the rendering resolution to n times the original resolution, such as 1.5 times or 2 times, when capturing images, to obtain higher-resolution images. Then, the electronic device uses an image super-resolution algorithm to optimize the higher-resolution image to further improve the image resolution. Experiments have shown that after processing with these two data enhancement methods, the edges of people and objects in the obtained high-resolution image are smoother and clearer. Optionally, the image super-resolution algorithm used by the electronic device may include but is not limited to the SRGAN (generative adversarial network super-resolution) algorithm.

[0147] As another feasible method, for low-resolution images, the electronic device can perform a downsampling strategy on them, and downsample the low-resolution images using a sampling coefficient of m. Optionally, the electronic device can perform multiple downsampling operations, for example, using a nearest neighbor interpolation downsampling with a sampling coefficient of 0.8, a linear downsampling of 0.85, and a linear sampling of 1.0 to process the low-resolution image, thereby obtaining an image with a lower resolution. Experiments have shown that this three-fold downsampling strategy can add more image quality problems to the original low-resolution image, such as jagged edges of characters, blurred distant views, etc.

[0148] Exemplarily, the electronic device may implement the above three-fold downsampling strategy in the following manner:

[0149] Assume that the resolution of the original low-resolution image is h×w=lr.size, the process of nearest neighbor interpolation downsampling with a sampling coefficient of 0.8 is lr=hr.resize([h×0.8, w×0.8], 'nearest'), the process of linear downsampling with a sampling coefficient of 0.85 is lr=hr.resize([h×0.85, w×0.85], 'linear'), and the process of linear sampling with a sampling coefficient of 1.0 is lr=hr.resize([h, w], 'linear').

[0150] It is understandable that the downsampling strategy implemented by the electronic device is not limited to the three downsamplings mentioned above, and other sampling coefficients may also be used to downsample the low-resolution image. This embodiment of the present application does not impose any restrictions on this.

[0151] As another feasible method, if the images in the training image set acquired by the electronic device contain text, the electronic device can also perform text separation processing on the images so that the images participating in the network model training process do not contain text, thereby avoiding the problem of blurred text in actual scenarios.

[0152] S302: Input the input image into the neural network model to be trained to obtain an output image.

[0153] Among them, the neural network model to be trained, that is, the network model that has not been trained yet, can include an initial neural network model or a neural network model in training (both can be referred to as neural network models for short), but the process of inputting the input image into the neural network model to be trained for processing is similar to the process of inputting the first image into the neural network model for processing in the above S103, that is, the input image is also convolved by the N convolution layers of the neural network model to obtain the output image.

[0154] In one embodiment, the electronic device may first perform an interpolation and amplification operation on the input image to obtain an amplified input image, and then perform image rearrangement (space to depth) on the amplified input image. After that, the electronic device may input the rearranged image into the neural network model for a convolution operation to obtain output results corresponding to multiple small images respectively. Finally, the output results corresponding to the multiple small images are subjected to the inverse operation of the above-mentioned image rearrangement (depth to space) to obtain the output image.

[0155] It is understandable that the output image may not be of high quality because the neural network model has not been trained yet. As the model training continues, the quality of the output image will become higher and higher.

[0156] S303: Calculate the loss between the output image and the supervision image, and adjust the network parameters of the neural network model according to the loss.

[0157] From the above description, it can be seen that the image quality of the output image is not high, that is, there is a certain error between the output image and the above-mentioned supervisory image (high-resolution image). Therefore, the electronic device can calculate the error (loss) between the output image and the supervisory image, and reversely adjust the network parameters of the neural network model according to the loss. For example, starting from the last convolution layer of the neural network model, adjust the network parameters of each convolution layer in turn until the first convolution layer is adjusted. Optionally, the electronic device can use a loss function to calculate the loss between the output image and the supervisory image.

[0158] In one embodiment, the electronic device can use one or more loss functions to calculate the loss between the output image and the supervision image to supervise the edges, details, and multiple dimensions of objects such as characters, ensuring that the game screen can be improved in game scenes such as character edges and distant views. For example, the loss functions used in the embodiments of the present application may include:

[0159] (1) Regression Loss Function of Output Result

[0160] The regression loss function of the output result can characterize the error between the overall output image and the overall supervision image. As can be seen from the above description, after the low-resolution input image is rearranged and input into the neural network model, the output results corresponding to multiple small images can be obtained. The output results can be multiple vector matrices; and the supervision image can also be inversely converted into multiple vector matrices of the same number. The regression loss function of the output result is to calculate the error between the multiple vector matrices corresponding to the output image and the multiple vector matrices corresponding to the supervision image. Optionally, the electronic device can calculate the regression loss function value according to Regression Loss = tf.generator.losses().

[0161] (2) Output feature loss function Feature Loss

[0162] The output feature loss function can characterize the error between the features of the output image and the features of the supervisory image. Therefore, the electronic device needs to extract the features of the output image and the features of the supervisory image. Exemplarily, the electronic device can input the output image and the supervisory image into the feature extraction network for feature extraction respectively. For example, the output image and the supervisory image can be input into the VGG19 network for feature extraction respectively. Among them, the VGG19 network contains 19 hidden layers, each layer is used to extract features of different depths. In this embodiment of the present application, the shallow features extracted from the 5th layer of the VGG19 network are preferably used to calculate the error. That is to say, the electronic device inputs the output image into the VGG19 network to extract the features of the 5th layer, and inputs the supervisory image into the VGG19 network to extract the features of the 5th layer, and calculates the error between the two features. Optionally, the electronic device can calculate the output feature loss function value according to Feature Loss = MeanAbsoluteError(VGG195(HR), VGG195(LR)), where HR is the supervisory image and LR is the output image.

[0163] (3) Sobel Loss based on the edge loss function of the Sobel operator

[0164] This edge loss function can characterize the error between edge features of a person, etc., in the output image and edge features of a person, etc., in the supervisory image. Therefore, the electronic device needs to further extract edge features of the output image and edge features of the supervisory image. For example, the electronic device can use a Sobel operator to extract edge features and then calculate the distance between the edge features.

[0165] Exemplarily, the process of extracting edge features using the Sobel operator may include:

[0166] input_gray=tf.image.rgb_to_grayscale(imgs);

[0167] Gx=tf.constant([[-1,0,1],[-2,0,2],[-1,0,1]], dtype='float32');

[0168] Gy=tf.constant([[-1,-2,-1],[0,0,0],[1,2,1]], dtype='float32');

[0169] Gx=tf.expand_dims(Gx, axis=2);

[0170] Gx=tf.expand_dims(Gx, axis=3);

[0171] Gy=tf.expand_dims(Gy, axis=2);

[0172] Gy=tf.expand_dims(Gy, axis=3);

[0173] x=tf.nn.conv2d(input_gray, Gx, strides=[1, 1, 1, 1], padding="SAME", name="hor");

[0174] y=tf.nn.conv2d(input_gray, Gy, strides=[1, 1, 1, 1], padding="SAME", name="vec");

[0175] G=tf.square(x)+tf.square(y);

[0176] The above process is to first determine the coordinates (x, y) of the edge points of the person in the image, and then connect the edge points to get the edge line. For the supervision image (HR), the Sobel feature G can be obtained. HR , the output image (LR) can get the Sobel feature G LR The electronic device can then HR and G LR Calculate the edge loss function value. Optionally, the electronic device can calculate the edge loss function value according to Sobel Loss = Mean Absolute Error (G HR , G LR ) calculates the marginal loss function value.

[0177] (4) Pixel Loss Function

[0178] The pixel-level loss function can represent the error between the pixel value of a pixel in the output image and the pixel value of a pixel in the supervision image, that is, the pixel value error of the pixel at the corresponding position in the output image and the supervision image. Optionally, the electronic device can calculate the pixel-level loss function value according to Pixel Loss = MeanAbsoluteError(HR, LR).

[0179] For the above four loss functions, the embodiments of the present application can use any one or more of them to calculate the loss between the output image and the supervisory image. If only one loss function is used to calculate the loss between the output image and the supervisory image, the electronic device can adjust the network parameters of the neural network model only according to the loss obtained. If multiple loss functions are used to calculate the loss between the output image and the supervisory image, the electronic device can weight the multiple losses calculated and adjust the network parameters of the neural network model according to the weighted losses, or the electronic device can adjust the network parameters of the neural network model in sequence according to the multiple losses.

[0180] It is understood that the training process of a neural network model typically requires multiple rounds of training and adjustment of network parameters. Therefore, the electronic device can repeatedly execute steps S302 to S303 based on the training image set until the training converges. The training convergence condition may include: the calculated loss between the output image and the supervisory image is less than a loss threshold, that is, the output image and the supervisory image are very close to each other and the error is negligible. At this point, the neural network model training can be considered to have converged.

[0181] It can also be understood that because the training process of the neural network model is computationally intensive and time-consuming, the training process can be executed on the server. After the server training is completed, the neural network model can be loaded into the GTX framework of the electronic device for the electronic device to subsequently perform the image processing process.

[0182] During the training process of the above-mentioned neural network model, the electronic device can use one or more loss functions to calculate the loss between the output image and the supervision image to supervise the edges, details, and multi-layer dimensions of characters, etc., thereby improving the processing accuracy of the neural network model and enabling higher-quality images to be obtained during subsequent image super-resolution processing.

[0183] For the above image processing process, combined with the above Figure 3 The software structure framework shown in the figure shows that the electronic device can realize the process of reducing the resolution of the image to be displayed obtained from the game application under the GTX framework, and then using the trained neural network model to perform image super-resolution processing on the low-resolution image to obtain a high-resolution image with the same resolution as the original image to be displayed and then display it. If combined with the QNN framework, the electronic device can Figure 5 The image rearrangement (space to depth) algorithm and the inverse operation (depth to space) algorithm of image rearrangement are integrated into libsr.so, and the image super-resolution algorithm (i.e. the convolution algorithm of the neural network model) is integrated into libsr_skel.so for GTX framework to call.

[0184] During the execution of libsr.so, since it involves the process of rearranging the first image, that is, dividing the first image into multiple small images, it involves the storage problem of multiple small images. For this, the embodiment of the present application uses the CDSP-Software-based L2FETCH instruction set, which can load the data of multiple small images in blocks, that is, perform L2Prefetch rewriting on the HVX vector corresponding to the first image according to the size of the available space in the cache, specify the initial Prefetch address, width, stride and height, which greatly reduces the memory stall caused by CDSP in data loading. In addition, in the implementation process, in addition to using the instructions of conventional calculations, you can also use The provided binary pixel permutation instructions and HVX 1024-bit SIMD instructions improve the flexibility and efficiency of CDSP vector processing. It is understood that the execution process of libsr.so can be executed on the CPU.

[0185] The execution of libsr_skel.so primarily involves the image super-resolution algorithm. As can be seen from the above description, this execution can be offloaded to the NPU to reduce GPU load. As can be understood, since the above-mentioned process of reducing the resolution of the displayed image consumes less power, this step can be retained on the GPU.

[0186] Based on this, combined with the above Figure 3 The software structure framework shown in FIG. 1 is provided in the embodiment of the present application. The embodiment of the present application also provides a flowchart of an image processing method, which mainly involves the signaling interaction between various modules, such as Figure 8 Specifically, it may include:

[0187] S11, the data capture module captures the image to be displayed from the game application.

[0188] The data capture module can capture the data of the image to be displayed from the game application through the capabilities of the GTX framework.

[0189] S12, the data capture module sends the image to be displayed to the scene analysis module.

[0190] S13, the scene analysis module performs text recognition on the image to be displayed.

[0191] S14: If it is recognized that the image to be displayed contains text, the scene analysis module separates the text from the image to be displayed to obtain the separated text and the image to be displayed containing only the image.

[0192] The method for the scene analysis module to perform text recognition and text separation on the image to be displayed can be referred to S202 in the above embodiment, and will not be repeated here.

[0193] S15, the scene analysis module sends the image to be displayed, which only includes the image, to the data capture module.

[0194] S16: The data capture module reduces the resolution of the image to be displayed to obtain a first image.

[0195] The data capture module may adopt a downsampling method to reduce the resolution of the image to be displayed to obtain the first image.

[0196] Optionally, after the data capture module reduces the resolution to obtain the first image, the resolution of the first image and the resolution of the image to be displayed can also be transmitted to the model determination module to determine the input resolution and output resolution of the neural network model, so that the subsequent model operation module can perform operations based on the input resolution and output resolution.

[0197] S17: The data capture module sends the first image to the operator optimization module.

[0198] S18: The operator optimization module performs an interpolation and magnification operation on the first image to obtain an enlarged first image.

[0199] S19: The operator optimization module calls libsr.so to rearrange the enlarged first image to obtain multiple small images.

[0200] Here, before using the neural network model to perform image super-resolution processing on the image to be displayed, the operator optimization module can call the aforementioned libsr.so to perform image permutation (space to depth) on the first image to optimize memory usage. After obtaining multiple small images after the permutation, subsequent image super-resolution processing can be performed.

[0201] S20, the operator optimization module sends multiple small images to the model operation module.

[0202] S21, the model operation module calls libsr_skel.so, uses the neural network model to perform image super-resolution processing on multiple small images, and obtains output results corresponding to the multiple small images.

[0203] Here, the model operation module can call libsr_skel.so to perform the image super-resolution processing process on the NPU. The process can be referred to the description of S103 above and will not be repeated here. The output results corresponding to the multiple small images can be multiple vector matrices.

[0204] S22, the model operation module sends the output results corresponding to the multiple small images to the operator optimization module.

[0205] S23 , the operator optimization module calls libsr.so to perform an inverse operation of image rearrangement on the output results corresponding to the multiple small images, to obtain a second image with the same resolution as the image to be displayed.

[0206] After obtaining the output results corresponding to multiple small images, if you want to obtain the result corresponding to the enlarged first image, you need to perform the inverse operation of the above-mentioned image rearrangement (depth to space) on the output results corresponding to the multiple small images respectively, that is, to obtain a high-resolution second image.

[0207] S24, the operator optimization module sends the second image to the image output module, and the scene analysis module sends the separated text to the image output module.

[0208] S25 , the image output module splices the second image with the separated text and displays the spliced ​​image.

[0209] After receiving the high-resolution second image and the separated text, the image output module can splice the two and display the spliced ​​image. For example, the image output module can call a display driver to display the spliced ​​image.

[0210] The above-mentioned image processing method first reduces the resolution of the image to be displayed obtained from the game application, and then uses the trained neural network model to perform image super-resolution processing on the low-resolution image to obtain a high-resolution image with the same resolution as the original image to be displayed, and then displays the high-resolution image. This can optimize the image to be displayed, improve key information such as image quality, details, and edges of objects and characters, and improve the picture quality in the game scene. In addition, the super-resolution processing process performed by the neural network model in the embodiment of the present application can be transferred to the NPU for processing, reducing the load on the GPU, that is, reducing the processing power consumption of the electronic device.

[0211] Based on the image processing method provided in the embodiments of this application, we also conducted corresponding experimental comparisons, comparing the operating effects of not using the image processing method provided in the embodiments of this application (Scenario 1), using the image processing method provided in the embodiments of this application (Scenario 2), and using the traditional MSR algorithm (Scenario 3) in a gaming scenario. The process of not using the image processing method provided in the embodiments of this application is the process of directly rendering and displaying the image to be displayed after obtaining the electronic device. Table 1 is a comparison of the power consumption of the electronic device when running Scenario 1, Scenario 2, and Scenario 3.

[0212] Table 1

[0213] Test Application Scenario Average frame rate Jitter rate Maximum number of dropped frames Number of freezes / hour Power consumption (mA) Application A Scenario 1 56.83 7.87％ 31 236 2160.29 Application A Scenario 2 57.1 3.93％ 26 58 1844.18 Application A Scenario 3 57.64 4.21％ 28 64 1906.22

[0214] Depend on Figure 1 It can be seen that when the image processing method provided in the embodiment of the present application is adopted (ie, scenario 2), the power consumption of the electronic device is the lowest, and the number of frame drops and pin jams is the least.

[0215] Figure 9 The display effect of the game screen when scene 1 and scene 2 are running is Figure 9 It can be seen that when the image processing method provided in the embodiment of the present application is adopted (ie, scenario 2), the display quality is significantly improved, especially the edge processing of characters is smoother.

[0216] In addition, an experiment was conducted in which the image super-resolution processing process was transferred to NPU processing while adopting the image processing method provided in an embodiment of the present application. The experimental results showed that the display quality remained basically unchanged, but the power consumption of the electronic device could be reduced by about 200mA (milliamperes).

[0217] The above describes in detail an example of the image processing method provided by the embodiment of the present application. It is understandable that, in order to implement the above functions, the electronic device includes hardware and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in a hardware or computer software driven hardware manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to be beyond the scope of the present application.

[0218] The embodiment of the present application can divide the functional modules of the electronic device according to the above method example. For example, each function can be divided into various functional modules, such as a detection unit, a processing unit, a display unit, etc., or two or more functions can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0219] It should be noted that all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0220] The electronic device provided in this embodiment is used to execute the above-mentioned image processing method, and thus can achieve the same effect as the above-mentioned implementation method.

[0221] When integrated, the electronic device may also include a processing module, a storage module, and a communication module. The processing module may be used to control and manage the operation of the electronic device. The storage module may be used to support the execution of program code and data stored in the electronic device. The communication module may be used to support communication between the electronic device and other devices.

[0222] The processing module may be a processor or a controller. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a digital signal processor (DSP) and a microprocessor, and so on. The storage module may be a memory. The communication module may specifically be a device that interacts with other electronic devices, such as a radio frequency circuit, a Bluetooth chip, or a Wi-Fi chip.

[0223] In one embodiment, when the processing module is a processor and the storage module is a memory, the electronic device involved in this embodiment may be a Figure 2 Device with the structure shown.

[0224] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the processor executes the image processing method of any of the above embodiments.

[0225] An embodiment of the present application further provides a computer program product. When the computer program product is run on a computer, the computer is caused to execute the above-mentioned related steps to implement the image processing method in the above-mentioned embodiment.

[0226] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor can execute the computer-executable instructions stored in the memory to enable the chip to execute the image processing method in the above-mentioned method embodiments.

[0227] Among them, the electronic device, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0228] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0229] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0230] Units described as separate components may or may not be physically separate, and components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0231] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0232] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0233] The above content is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. An image processing method, the method being executed by an electronic device, characterized in that: The electronic device is built with a GTX framework, which includes a data capture module, a scene analysis module, an operator optimization module, a model operation module, and an image output module. The electronic device is also built with an AP library and a CP library, and the AP library and the CP library integrate an algorithm of a neural network model for performing image super-resolution processing on an image. The method includes: The data capture module acquires an image to be displayed on the electronic device, where the resolution of the image to be displayed is a first resolution; The data capture module sends the image to be displayed to the scene analysis module; The scene analysis module performs text recognition on the image to be displayed; If it is recognized that the image to be displayed contains text, the scene analysis module separates the text from the image to be displayed to obtain the separated text and the image to be displayed containing only the image; The scene analysis module sends the image to be displayed containing only the image to the data capture module; The data capture module performs resolution reduction processing on the image to be displayed that only contains an image to obtain a first image, wherein the resolution of the first image is a second resolution, and the first resolution is greater than the second resolution; The data capture module sends the first image to the operator optimization module; The operator optimization module performs an interpolation and amplification operation on the first image to obtain an amplified first image; The operator optimization module calls the AP library and performs an image rearrangement operation on the CPU to obtain a plurality of small images arranged according to image depth; The operator optimization module sends a plurality of small images to the model operation module; The model operation module calls the CP library and uses a neural network model on the embedded neural network processor NPU to perform image super-resolution processing on the multiple small images to obtain output results corresponding to the multiple small images respectively; The model operation module sends output results corresponding to multiple small images to the operator optimization module; The operator optimization module calls the AP library and performs an inverse operation of image rearrangement on the output results corresponding to the plurality of small images on the CPU to obtain a second image; The operator optimization module sends the second image to the image output module, and the scene analysis module sends the separated text to the image output module; The image output module splices the second image and the separated text to obtain a second image containing text, and displays the second image containing text, wherein the resolution of the second image is the first resolution, and the picture quality of the second image is better than the picture quality of the image to be displayed; Before performing image super-resolution processing on the plurality of small images using a neural network model, the method further includes: Acquire a training image set, the training image set including an input image and a supervisory image, wherein a resolution of the supervisory image is greater than a resolution of the input image; Inputting the input image into the neural network model to be trained to obtain an output image; Calculating a loss between the output image and the supervision image, and adjusting network parameters of the neural network model to be trained according to the loss to train the neural network model, where the loss includes a first loss, a second loss, a third loss, and a fourth loss; The first loss represents the error between multiple vector matrices corresponding to the output image and multiple vector matrices corresponding to the supervision image, the second loss represents the error between the features of the output image and the features of the supervision image, the third loss represents the error between the edge features of the objects contained in the output image and the edge features of the objects contained in the supervision image, and the fourth loss represents the error between the pixel values ​​of the pixels in the output image and the pixel values ​​of the pixels in the supervision image.

2. The method according to claim 1, characterized in that After acquiring the training image set, the method further includes: Performing image super-resolution processing on the supervision image to obtain a supervision image with improved resolution; The input image is subjected to resolution reduction processing to obtain an input image with reduced resolution.

3. The method according to claim 1 or 2, characterized in that The input image includes a plurality of images with different resolutions.

4. The method according to any one of claims 1 to 3, characterized in that The neural network model is a network model including 7 convolutional layers.

5. The method according to any one of claims 1 to 4, characterized in that The image to be displayed is an image of a game application running on the electronic device.

6. An electronic device, characterized in that: include: one or more processors; one or more memories; The memory stores one or more programs, and when the one or more programs are executed by the processor, the electronic device executes the method according to any one of claims 1 to 5.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the processor calls instructions to enable the electronic device to execute the method according to any one of claims 1 to 5.

Images