Image processing method and electronic device

Abstract

The embodiment of the present application provides an image processing method and an electronic device, which is executed by an electronic device, including: obtaining a current frame image to be displayed by the electronic device, a historical frame image of a first resolution, a first vector, and image depth information corresponding to the current frame image, wherein the resolution of the historical frame image is greater than the resolution of the current frame image, and the first vector represents the displacement between the position of the same pixel point in the current frame image and the position in the historical frame image; inputting the current frame image, the historical frame image, the first vector, and the image depth information into a neural network model for image super-resolution processing, obtaining a current frame image of a first resolution, and displaying the current frame image of the first resolution. In this method, while improving the image resolution, the continuity between the current frame image and the historical frame image can also be taken into account, reducing problems such as flickering and ghosting of the game screen, so that the picture quality in the game scene is better.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method and electronic device. Background Art

[0002] Currently, the functions of wireless electronic devices are becoming more and more abundant. For example, for mobile electronic devices such as mobile phones and tablets, in addition to meeting users' daily communication and office needs, they can also run real-time online games to improve users' experience.

[0003] However, for online games with strong real-time requirements, they have high requirements for key information such as the image quality, details, and edges of objects and characters. If the game image quality is poor, it may seriously affect the user's gaming experience. Therefore, it is necessary to study how to improve the image quality of mobile electronic devices in game scenarios. Summary of the invention

[0004] The present application provides an image processing method and an electronic device, which can greatly improve the picture quality of the electronic device in a game scenario.

[0005] In a first aspect, the present application provides an image processing method, which is executed by an electronic device, and includes: obtaining a current frame image to be displayed by the electronic device, a historical frame image of a first resolution, a first vector, and image depth information corresponding to the current frame image, wherein the resolution of the historical frame image is greater than the resolution of the current frame image, and the first vector represents the displacement between the position of the same pixel point in the current frame image and the position in the historical frame image; inputting the current frame image, the historical frame image, the first vector, and the image depth information into a neural network model for image super-resolution processing to obtain a current frame image of the first resolution, and displaying the current frame image of the first resolution.

[0006] Among them, the current frame image to be displayed can be a screen to be displayed of a game application running on the electronic device. For example, when an application is running on the electronic device, the current frame image to be displayed is the display screen of the application. Exemplarily, the current image to be displayed can be a screen to be displayed of a game application running on the electronic device. The historical frame image can be an image displayed before the current frame image. In the present application, only the previous frame image of the current frame image can be used as the historical frame image to reduce the amount of input data of the neural network model. The first vector is the motion vector, that is, the motion vector between the current frame image and the historical frame image, which is used to characterize the displacement between the position of the same pixel in the current frame image and the position in the historical frame image. The image depth information characterizes the depth information of the pixel in the current frame image. Optionally, in the present application, the electronic device can directly obtain the above-mentioned motion vector and image depth information from the rendering engine of the game application.

[0007] After acquiring the current frame image, the historical frame image, the first vector and the image depth information, the electronic device can input this information into the trained neural network model for image super-resolution processing to obtain a high-resolution current frame image. Here, the neural network model used in the present application can realize the conversion process from a low-resolution image to a high-resolution image, and the neural network model focuses on processing the details of the objects contained in the image during training. For example, the features of the objects or characters contained in the image can be refined to obtain a better image quality. In addition, the neural network model can receive input of multiple frames of information, that is, the electronic device can input the above-mentioned multiple frames of information into the neural network model, and the neural network model performs image super-resolution processing to obtain a high-resolution current frame image.

[0008] In the above implementation, the electronic device can perform image super-resolution processing on the current frame image based on multiple frame information such as high-resolution historical frame images, current frame images, motion vectors between historical frame images and current frame images, and image depth information to obtain a high-resolution current frame image. After this process, the display image of the game application improves the resolution while also taking into account the continuity between the current frame image and the historical frame image, which can reduce problems such as flickering and ghosting of the game screen, making the picture quality in the game scene better.

[0009] In combination with the first aspect, in some implementations of the first aspect, the current frame image, the historical frame image, the first vector and the image depth information are input into a neural network model for image super-resolution processing to obtain a current frame image of a first resolution, including: obtaining a spliced ​​image based on the current frame image, the historical frame image and the first vector; and inputting the spliced ​​image and the image depth information into a neural network model for image super-resolution processing to obtain a current frame image of a first resolution.

[0010] Among them, when the electronic device uses the neural network model for processing, it can first align and splice the current frame image and the historical frame image, and then use the spliced ​​image and image depth information to perform image super-resolution processing to obtain the current frame image of the first resolution.

[0011] In one implementation, the above-mentioned obtaining a spliced ​​image based on the current frame image, the historical frame image and the first vector includes: determining a predicted image for the current frame image based on the historical frame image and the first vector; aligning and splicing the predicted image and the current frame image to obtain a spliced ​​image.

[0012] Among them, because the first vector represents the displacement between the position of the same pixel point in the current frame image and the position in the historical frame image, the electronic device can predict a current frame image based on the historical frame image and the motion vector, which is referred to as a predicted image. Then, the electronic device can align and splice the predicted image and the current frame image to obtain a spliced ​​image. For example, the electronic device can align and splice the predicted image and the current frame image in the channel dimension.

[0013] In another implementation, since the resolution of the high-resolution historical frame image is greater than the resolution of the current frame image, the resolution of the predicted image predicted based on the historical frame image and the motion vector may also be greater than the resolution of the current frame image. Therefore, before aligning and splicing the predicted image and the current frame image, the method further includes: performing an interpolation and magnification operation on the current frame image to obtain an enlarged current frame image, and the size of the enlarged current frame image is the same as the size of the predicted image.

[0014] Correspondingly, aligning and splicing the predicted image with the current frame image to obtain a spliced ​​image includes: aligning and splicing the predicted image with the enlarged current frame image to obtain a spliced ​​image.

[0015] That is to say, the electronic device can first enlarge the size of the current frame image to the same size as the predicted image, and then align and splice the two to ensure that the two images to be aligned have the same size, thereby improving the accuracy of image alignment.

[0016] Taking into account actual applications, pixels in the image may be blocked, or the pixel values ​​of the same pixel may be inconsistent due to factors such as lighting, or pixel rendering may be lost during the image rendering process, etc. These abnormal situations may cause the pixel value errors in the predicted image or the current frame image after interpolation and amplification to be too large. Therefore, in the present application, the electronic device can perform error statistics on the predicted image and the current frame image after interpolation and amplification, and correct the pixel values ​​with excessive errors, and then perform splicing after correction.

[0017] In combination with the first aspect, in some implementations of the first aspect, after obtaining the enlarged current frame image, the above method also includes: calculating the pixel value error between each pixel point in the predicted image and the enlarged current frame image; if the pixel value error corresponding to the first pixel point is greater than the error threshold, then correcting the pixel value of the first pixel point in the predicted image and / or the pixel value of the first pixel point in the enlarged current frame image, the first pixel point is any pixel point in the predicted image and the enlarged current frame image.

[0018] Optionally, the electronic device may use a filtering error method to calculate the pixel value error between the predicted image and the pixel point in the enlarged current frame, and may determine the correction value of the first pixel point based on the pixel values ​​of the surrounding pixels based on the clamp operation to correct the pixel value of the first pixel point. This can improve the accuracy of the obtained stitched image, and further improve the accuracy of the subsequent high-resolution current frame image.

[0019] In combination with the first aspect, in some implementations of the first aspect, the above-mentioned inputting the stitched image and image depth information into the neural network model for image super-resolution processing to obtain the current frame image of the first resolution includes: weighting the stitched image based on the image depth information; inputting the stitched image after weight allocation into the neural network model for image super-resolution processing to obtain the current frame image of the first resolution.

[0020] Among them, for a frame of image, the pixel changes at the edges of objects (such as people, objects, etc.) contained in the image or the overlap of objects are usually more obvious, and the depth value of the pixels at the corresponding positions also changes greatly. Then the subsequent neural network model needs to focus on this part of the image when performing image super-resolution processing to optimize the image quality problems that may occur in this part. Therefore, the electronic device can assign weights to the spliced ​​images based on the image depth information, assigning larger weights to the pixels with larger corresponding depth value changes in the spliced ​​images, and assigning smaller weights to the pixels with smaller corresponding depth value changes in the spliced ​​images. Optionally, the electronic device can use an attention mechanism to assign weights to the spliced ​​images based on the image depth information. As a result, the electronic device can optimize the edge features of the objects or characters contained in the image, and further improve the picture quality of the game application.

[0021] In combination with the first aspect, in some implementations of the first aspect, after obtaining the stitched image, the method further includes: rearranging the stitched image to obtain a plurality of small images arranged according to image depth;

[0022] Correspondingly, the weight allocation to the stitched image based on the image depth information includes: weight allocation to the plurality of small images based on the image depth information.

[0023] In actual scenarios, due to the memory limitation of electronic devices, if the stitched images are directly stored and subsequently operated, insufficient memory may occur. Therefore, in order to reasonably allocate memory, after obtaining the stitched images, the electronic device can also rearrange the stitched images (space to depth) to obtain multiple small images arranged according to the image depth, and then weight the multiple small images after the image rearrangement. After that, the electronic device can input the multiple small images after the weight allocation into the neural network model for image super-resolution processing to obtain the output results corresponding to the multiple small images, for example, the output results are multiple vector matrices; if the result corresponding to the stitched image is to be obtained, the inverse operation of the above-mentioned image rearrangement (depth to space) is required for the output results corresponding to the multiple small images, that is, to obtain a high-resolution current frame image. In this way, the electronic device can reasonably allocate memory and reduce the occurrence of memory problems.

[0024] In combination with the first aspect, in some implementations of the first aspect, before the current frame image, the historical frame image, the first vector and the image depth information are input into the neural network model for image super-resolution processing, the method further includes: obtaining a training image set, the training image set including input image information and a supervisory image, the input image information including a first image of a second resolution, a second image of the first resolution, a second vector and image depth information corresponding to the first image, the second vector representing the displacement between the position of the same pixel point in the first image and the position in the second image, the supervisory image including the first image of the first resolution, the first resolution being greater than the second resolution; inputting the input image information into the neural network model to be trained to obtain an output image; calculating the loss between the output image and the supervisory image, and adjusting the network parameters of the neural network model to be trained according to the loss to train the neural network model.

[0025] It can be seen from the above description that the neural network model used by the electronic device is a trained network model, so before using the neural network model, it needs to be trained to improve the accuracy of the processing results.

[0026] In this implementation, the electronic device can obtain a large number of training image sets, wherein the input image information is used to input into the neural network model for processing, and the supervision image is used to supervise the difference between the output result of the neural network model and the supervision image, thereby supervising the learning and training process of the neural network model. In one implementation, the input image information may include a low-resolution first image and a high-resolution second image, as well as a motion vector and image depth information between the two. The supervision image may include a high-resolution first image, and the high-resolution image is used as a supervision signal to calculate the difference between the output image of the neural network model and the high-resolution image to train the neural network model.

[0027] In the process of model training, the loss between the output image of the neural network model and the supervision image is usually calculated, and the network parameters of the neural network model are continuously adjusted according to the loss. The loss function is usually also varied. In the present application, in combination with the first aspect, in some implementations of the first aspect, the above-mentioned loss includes at least one of the first loss, the second loss, the third loss and the fourth loss.

[0028] The first loss represents the error between multiple vector matrices corresponding to the output image and multiple vector matrices corresponding to the supervision image, the second loss represents the error between the features of the output image and the features of the supervision image, the third loss represents the error between the edge features of the objects contained in the output image and the edge features of the objects contained in the supervision image, and the fourth loss represents the error between the pixel values ​​of the pixels in the output image and the pixel values ​​of the pixels in the supervision image.

[0029] Among them, the electronic device can use one or more loss functions to calculate the loss between the output image and the supervision image to supervise the edges, details, and multi-dimensionality of objects such as characters, ensuring that the game screen can be improved in game scenes such as character edges and distant views.

[0030] The first loss can be the regression loss function value of the output result. Optionally, the electronic device can calculate the regression loss function value according to Regression Loss = tf.generator.losses(). The second loss can be the output feature loss function value. Optionally, the electronic device can input the output image and the supervision image into the feature extraction network for feature extraction, and calculate the loss based on the extracted features. The third loss can be the edge loss function value. Optionally, it can be the edge loss function value based on the Sobel operator. The fourth loss can be a pixel-level loss function value, that is, for the output image and the supervision image, the pixel value error of the pixel points at the corresponding position in the image.

[0031] In this implementation, the electronic device calculates the loss between the output image and the supervision image by using one or more loss functions to supervise the edges, details, and multi-layer dimensions of characters, etc., thereby improving the processing accuracy of the neural network model, so that higher quality images can be obtained when subsequent image super-resolution processing is performed.

[0032] In combination with the first aspect, in some implementations of the first aspect, after obtaining the training image set, the above method also includes: performing image super-resolution processing on the supervision image to obtain a supervision image with improved resolution; and performing resolution reduction processing on the first image of the second resolution to obtain the first image with reduced resolution.

[0033] In order to improve the neural network model's ability to handle more image quality problems and improve the accuracy of the neural network model's output results, electronic devices can perform data enhancement preprocessing on the training image set, such as reducing the clarity of the input image (low-resolution image) to make it have more image quality problems, and improving the clarity of the supervision image (high-resolution image) to ensure that the neural network model can ultimately output more accurate results.

[0034] Therefore, the electronic device can perform image super-resolution processing on the supervision image to improve the resolution of the supervision image, and the edges of people and objects in the obtained supervision image are smoother and clearer. The input image can also be processed to reduce the resolution, such as downsampling, to obtain an input image with a lower resolution.

[0035] In combination with the first aspect, in some implementations of the first aspect, before the current frame image, the historical frame images, the first vector and the image depth information are input into the neural network model for image super-resolution processing, the method further includes: performing text recognition on the current frame image; if it is recognized that the current frame image contains text, performing text separation on the current frame image to obtain the separated text and the current frame image that only contains the image.

[0036] Because in some scenes (such as game scenes), the current frame image may contain text. If the current frame image with text is subjected to super-resolution processing, it may cause text distortion and other phenomena. Therefore, the present application can also separate the text before processing the current frame image. Optionally, the electronic device can use a text recognition algorithm to identify the current frame image to determine whether the current frame image contains text; when the electronic device recognizes that the current frame image contains text, the text can be separated, for example, using a segmentation algorithm to segment the text to obtain the separated text and the image to be displayed that only contains the image. It can be understood that the historical frame image may also contain text. Therefore, when executing this step, the electronic device can also perform text recognition on the historical frame image at the same time to separate the text therein.

[0037] Correspondingly, in the case of separating text, after the current frame image, the historical frame image, the first vector and the image depth information are input into the neural network model for image super-resolution processing to obtain the current frame image of the first resolution, the method further includes: splicing the current frame image of the first resolution and the separated text to obtain the current frame image containing the text.

[0038] In this implementation, since the text in the current frame image does not participate in the image processing process, the displayed text is still the original text, which does not affect the display effect of the text, and further improves the picture quality of the final displayed image.

[0039] In combination with the first aspect, in some implementations of the first aspect, the electronic device includes an embedded neural network processor (neural-network processing unit, NPU), and the current frame image, historical frame images, the first vector and image depth information are input into the neural network model for image super-resolution processing, including: inputting the current frame image, historical frame images, the first vector and image depth information into the neural network model through the NPU for image super-resolution processing.

[0040] Among them, since the traditional image rendering process is executed on the graphics processing unit (GPU), this undoubtedly increases the load of the GPU. In this regard, the present application transfers the image super-resolution processing process performed by the neural network model to the NPU for processing, thereby reducing the load of the GPU, that is, reducing the processing power consumption of the electronic device.

[0041] In addition, the electronic device is also built with GTX (GPU turbo X) framework and QNN (qualcomm neural network) framework, build an AP library (libsr.so) and a CP library (libsr_skel.so) in the QNN framework, the relevant algorithms of the neural network model for image super-resolution processing can be integrated in the AP library and the CP library, and the calculation of the CP library can be transferred to the NPU for processing for GTX framework to call.

[0042] In a second aspect, the present application provides a device, which is included in an electronic device, and the device has the function of implementing the electronic device behavior in the above-mentioned first aspect and the possible implementation of the above-mentioned first aspect. The function can be implemented by hardware, or it can be implemented by hardware executing corresponding software. The hardware or software includes one or more modules or units corresponding to the above-mentioned functions. For example, a receiving module or unit, a processing module or unit, etc.

[0043] In a third aspect, the present application provides an electronic device, comprising: a processor, a memory, and an interface; the processor, the memory, and the interface cooperate with each other so that the electronic device executes any one of the methods in the technical solution of the first aspect.

[0044] In a fourth aspect, the present application provides a chip, including a processor, wherein the processor is configured to read and execute a computer program stored in a memory to call instructions so that an electronic device executes the method in the first aspect and any possible implementation thereof.

[0045] Optionally, the chip also includes a memory, and the memory is connected to the processor via a circuit or wire.

[0046] Further optionally, the chip also includes a communication interface.

[0047] In a fifth aspect, the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the processor calls instructions so that the electronic device executes any one of the methods in the technical solution of the first aspect.

[0048] In a sixth aspect, the present application provides a computer program product, the computer program product comprising: a computer program code, when the computer program code runs on an electronic device, the electronic device executes any one of the methods in the technical solution of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0050] Figure 2 is a software structure block diagram of an electronic device provided in an embodiment of the present application;

[0051] Figure 3 is a flowchart of an image processing method provided in an embodiment of the present application;

[0052] Figure 4 is a schematic diagram of a comparison of a current frame image, a historical frame image and a motion vector provided in an embodiment of the present application;

[0053] Figure 5 is a schematic diagram of a process of an image processing method provided in an embodiment of the present application;

[0054] Figure 6 is a schematic diagram of a pixel value correction process provided in an embodiment of the present application;

[0055] Figure 7 is a flowchart of another image processing method provided in an embodiment of the present application;

[0056] Figure 8 is a schematic diagram of a training process of a neural network model provided in an embodiment of the present application;

[0057] Fig. 9 is a flowchart of another image processing method provided in an embodiment of the present application;

[0058] Fig.10 This is a comparison chart of the display effects of the game screen when running in different scenarios provided by the embodiment of the present application. DETAILED DESCRIPTION

[0059] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0060] In the following, the terms "first", "second", and "third" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first", "second", and "third" may explicitly or implicitly include one or more of the features.

[0061] Traditional online games are usually run on electronic devices with strong computing power, such as personal computers (PCs) and ultra-mobile personal computers (UMPCs). However, with the continuous development of electronic technology, the performance of mobile electronic devices has been greatly improved. Online games with high real-time requirements can also be run on mobile electronic devices, such as mobile phones, tablet computers and other electronic devices.

[0062] When running game applications on electronic devices such as mobile phones and tablets, in order to improve the quality of the displayed game screen, related technologies propose that the single-frame game screen can be optimized, such as increasing the resolution or optimizing local details. However, in the game scene, there is usually a certain continuity between the game screen of the current frame and the game screen of the historical frame. If this continuity is not considered during the optimization process, problems such as flickering and ghosting of the game screen may occur, which has a negative impact on the user experience.

[0063] In view of this, an embodiment of the present application provides an image processing method, which can use a trained neural network (NN) model based on high-resolution historical frame images, current frame images, motion vectors between historical frame images and current frame images, and image depth information (depth buffer) and other multi-frame information to perform image super-resolution processing (also referred to as super-resolution processing) on ​​the current frame image to obtain a high-resolution current frame image, and then display the high-resolution current frame rate image. After the image processing method of the embodiment of the present application, the display image of the game application improves the resolution while also taking into account the continuity between the current frame image and the historical frame image, which can reduce problems such as flickering and ghosting of the game screen, making the picture quality in the game scene better.

[0064] As is known to all, the neural network model used for image super-resolution usually refers to the use of a series of convolution operations and nonlinear mappings, as well as different functional structures composed of convolution operations and nonlinear mappings, to complete the super-resolution operation. According to the different characteristics of the convolution operations at different stages, the different functional structures can be divided into feature extraction modules, nonlinear mapping modules, attention modules, alignment modules, propagation modules, reconnection modules, reconstruction modules, upsampling modules, etc., and their essence is to determine a mapping relationship so that the super-resolution image generated by the low-resolution image is as similar as possible to the original high-resolution image. At present, there are some neural network models for image super-resolution, which can consider the continuity between the current frame image and the historical frame image, so that the neural network model outputs a high-resolution image for the input of multiple frames of images, but most of these neural network models require multiple (for example, more than 5) historical frame images as input, which will result in a large amount of calculation and high latency of the model, and cannot be directly applied to mobile electronic devices.

[0065] In addition, when considering the continuity between the current frame image and the historical frame image, the motion vector between the historical frame image and the current frame image is more important. The more accurate the motion vector is, the more accurate the high-resolution current frame image is. When determining the motion vector between the historical frame image and the current frame image, some current methods estimate the motion vector based on continuous historical frame images, and then estimate the high-resolution current frame image based on the motion vector, which will undoubtedly produce estimation errors; some other methods do not determine the above motion vector, and directly perform super-resolution prediction through the historical frame image and the current frame image, which will also produce prediction errors.

[0066] The neural network model provided in the embodiment of the present application is a lightweight network. When inputting the neural network model, accurate motion vectors can be directly obtained from the game application, and the motion vectors, image depth information, current frame images, and a high-resolution historical frame image can be input into the neural network model, thereby reducing the input amount of the neural network model, and can be better oriented to the game scene, and can be applied to mobile electronic devices. At the same time, by directly obtaining accurate motion vectors, prediction errors can also be reduced, and the accuracy of the output high-resolution current frame image can be improved. It should be noted that the image processing method provided in the embodiment of the present application can be applied to mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, laptops, netbooks, personal digital assistants (PDAs) and other mobile electronic devices with the above-mentioned super-resolution processing capabilities. The embodiment of the present application does not impose any restrictions on the specific types of electronic devices.

[0067] Furthermore, the embodiments of the present application also propose to transfer the image super-resolution processing process performed by the neural network model to the NPU for processing, thereby reducing the load of the GPU, that is, reducing the processing power consumption of the electronic device.

[0068] For example, Figure 1 1 is a schematic diagram of the structure of an electronic device 100 provided in an embodiment of the present application. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0069] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a GPU, an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or an NPU, etc. Different processing units may be independent devices or integrated into one or more processors.

[0070] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0071] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0072] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0073] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0074] The charging management module 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the electronic device 100. While the charging management module 140 is charging the battery 142, it may also power the electronic device through the power management module 141.

[0075] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the external memory, the display screen 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0076] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0077] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Figure 1 The structures of antenna 1 and antenna 2 in the electronic device 100 are only examples. Each antenna in the electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of the antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0078] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0079] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0080] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0081] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0082] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0083] NPU is a neural network computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of the electronic device 100 can be realized, such as: image recognition, face recognition, voice recognition, text understanding, etc.

[0084] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0085] The internal memory 121 can be used to store computer executable program codes, and the executable program codes include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0086] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0087] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The electronic device 100 may receive key input and generate key signal input related to user settings and function control of the electronic device 100.

[0088] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 194, motor 191 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0089] Indicator 192 may be an indicator light, which may be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0090] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0091] The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system of the layered architecture as an example to exemplify the software structure of the electronic device 100.

[0092] Figure 2 1 is a software structure diagram of the electronic device 100 of the embodiment of the present application. The layered architecture divides the software into several layers, each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom, namely, the application layer, the application framework layer, the Android runtime (Android runtime) and the system library, and the kernel layer.

[0093] like Figure 2 As shown, the application layer can include a series of application packages. The application package can include games, cameras, gallery, calendar, calls, maps, navigation, WLAN, Bluetooth, music, video, short messages and other applications. For the convenience of illustration, Figure 2 Only game applications are used as an example.

[0094] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0095] like Figure 2 As shown, the application framework layer can build a GTX (GPU turbo X) framework, which can usually perform game logic processing and global optimization rendering processing. In one embodiment, the GTX framework may include a data capture module, a model loading module, a rendering processing module, a splicing module, an operator optimization module, an attention module, a model operation module, and an image output module.

[0096] Among them, the data capture module can capture the data of the current frame image to be displayed from the game application, and convert the image format, etc. The model loading module can initialize the neural network model, including but not limited to model loading, model compilation, operation check and memory configuration. The rendering processing module can select the rendering method and change the window resolution after loading the neural network model. The operator optimization module can perform memory operator optimization before inputting the current frame image into the neural network model, such as rearranging the image to reduce memory usage. The model operation module can use the neural network model to perform image super-resolution operations based on the input current frame image, high-resolution historical frame images, motion vectors and image depth information to obtain a high-resolution current frame image. The image output module can be used to display the output high-resolution image.

[0097] Before the model operation module performs the image super-resolution operation, the splicing module can also splice the historical frame image based on the high resolution, the motion vector and the current frame image. The attention module can also assign weights to the obtained splicing module.

[0098] In addition, the above application framework layer can also include window manager, content provider, view system, phone manager, resource manager, notification manager, etc. ( Figure 2 Not shown yet).

[0099] The window manager is used to manage window programs. The window manager can obtain the size of the display screen, determine whether there is a status bar, lock the screen, capture the screen, etc. The content provider is used to store and obtain data and make the data accessible to applications. The data may include video, images, audio, calls made and received, browsing history and bookmarks, phone books, etc. The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build applications. The display interface can be composed of one or more views. For example, a display interface including a text message notification icon may include a view for displaying text and a view for displaying pictures. The phone manager is used to provide communication functions for the electronic device 100. For example, the management of call status (including connected, hung up, etc.). The resource manager provides various resources for applications, such as localized strings, icons, pictures, layout files, video files, etc. The notification manager enables applications to display notification information in the status bar, which can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the top status bar of the system in the form of an icon or scroll bar text, such as a notification of an application running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is prompted in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.

[0100] Android runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.

[0101] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0102] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0103] The system library may include multiple functional modules, such as surface manager, media libraries, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0104] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0105] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0106] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0107] A 2D graphics engine is a drawing engine for 2D drawings.

[0108] The kernel layer is the layer between hardware and software. The kernel layer includes at least display driver, camera driver, audio driver, etc. For example, the display driver can be used to receive image data transmitted by the game application to display the corresponding image screen.

[0109] Based on the above GTX framework, the embodiment of the present application can realize the process of using the trained neural network model to perform image super-resolution processing on the current frame image obtained from the game application to obtain a high-resolution image and then display it. In addition, if the image super-resolution processing process performed by the neural network model is to be transferred to the NPU for processing, the embodiment of the present application can also use The QNN framework platform transfers the processing process to reduce the GPU load. The following is a brief introduction to the performance advantages of the QNN framework platform:

[0110] First of all, compared with other artificial intelligence (AI) frameworks, such as the SNPE (snapdragon neural processing engine) framework, QNN can reduce memory copies, use ion buffers (ION buffer) and register them to the hardware acceleration unit CDSP (compute DSP), which can reduce performance overhead compared to ordinary buffers. For example, experiments have shown that the performance overhead reduced by 1MB is about 0.6mA (milliamperes). Secondly, QNN can increase the usage ratio of CDSP, reduce CPU power consumption compared to GPU, and increase cache utilization. Finally, QNN can also optimize the processing operator HVX (Hexagon vector extension, i.e. Hexagon-hexagon, Vector-vector, Extensions-extension) before and after the neural network model processing through the single instruction multiple data (SIMD) capability of CDSP, and improve the operating efficiency of the processing operator.

[0111] Therefore, the embodiment of the present application uses QNN as the basic framework, and uses the latest Hexagon SDK version to build an AP library (libsr.so) and a CP library (libsr_skel.so). The AP library is responsible for the external interface, and can perform model parsing and schedule CDSP libraries, etc. (NDK compilation tools can be used here), and the CP library is responsible for the specific super-resolution algorithm implementation, including algorithm logic and related operators, etc. (Hexagon Tool compilation tools can be used here), and the AP library and the CP library can be scheduled through the IDL interface (such as fastRPC). Among them, the relevant algorithms of the neural network model for super-resolution processing of images can be integrated in the AP library and the CP library, and the operations of the CP library can be transferred to the NPU for processing for GTX framework calls.

[0112] Based on the above-mentioned GTX framework and QNN framework, the image processing method provided by the embodiment of the present application will be described in detail below. For ease of understanding, the following embodiments of the present application will be described with Figure 1 and Figure 2 Taking the electronic device with the structure shown as an example, the image processing method provided in the embodiment of the present application is specifically described in combination with the accompanying drawings and application scenarios. Figure 3 : is a flowchart of an image processing method provided in an embodiment of the present application, and the method may include:

[0113] S101, obtaining a current frame image to be displayed by an electronic device, high-resolution historical frame images, motion vectors, and image depth information.

[0114] The current frame image to be displayed may be a screen to be displayed of a game application running on the electronic device. Since the screen of the game application changes in real time, the electronic device may also obtain the image to be displayed at each moment in real time. For example, the current frame image to be displayed is obtained, and the resolution of the image is 720p. Optionally, the electronic device may capture the current frame image to be displayed of the game application in real time through the GTX framework.

[0115] The historical frame image may be an image displayed before the current frame image. In the embodiment of the present application, only the previous frame image of the current frame image may be used as the historical frame image. The high-resolution historical frame image may be a high-resolution image obtained after the image super-resolution processing is performed on the historical frame image using the neural network model at the last moment. It can be understood that when the neural network model is used for processing at the last moment, the input is the low-resolution historical frame image, the previous high-resolution frame image corresponding to the historical frame image, the motion vector between the two, and the image depth information; after obtaining the high-resolution historical frame image, the electronic device may cache it to the shared memory, so that when the image super-resolution processing is performed on the current frame image, the corresponding high-resolution historical frame image is obtained from the shared memory. It can be understood that the resolution of the current frame image obtained by the electronic device is less than the resolution of the high-resolution historical frame image. The image processing method of the embodiment of the present application is to increase the resolution of the current frame image to the resolution corresponding to the high-resolution historical frame image. For the convenience of description, the high-resolution historical frame image may also be referred to as the historical frame image below.

[0116] The motion vector is the motion vector between the current frame image and the historical frame image, and is used to characterize the displacement between the position of the same pixel point in the current frame image and the position in the historical frame image. In an embodiment of the present application, the electronic device can directly obtain the motion vector from the rendering engine of the game application, without the need to estimate the motion vector, thereby reducing the estimation error; and in actual application, when the electronic device obtains the motion vector from the rendering engine of the game application, it can send a request to the rendering engine to obtain the motion vector of the first size, and the rendering engine can draw the motion vector of the first size according to the request to return it to the electronic device, where the first size can be the same as the size of the historical frame image and / or the current frame image.

[0117] Image depth information represents the depth information of the pixel in the current frame image. Depth can be understood as the distance of the pixel from the camera in the 3D world. The larger the depth value, the farther the pixel is from the camera. In the embodiment of the present application, the electronic device can also directly obtain the image depth information from the rendering engine of the game application.

[0118] It should be understood that the image processing method provided in the embodiment of the present application can be applied not only in the running process of the game application, but also in other scenarios, such as video playback scenarios, video conferencing scenarios, etc., which have high requirements on the quality of the display picture.

[0119] S102, obtaining a spliced ​​image according to the current frame image, high-resolution historical frame images and motion vectors.

[0120] Since the motion vector represents the displacement between the position of the same pixel point in the current frame image and the position in the historical frame image, the electronic device can predict a current frame image based on the historical frame image and the motion vector, which is referred to as a predicted image. Then, the electronic device can align and splice the predicted image with the current frame image to obtain a spliced ​​image.

[0121] In an achievable manner, the motion vector may be a vector matrix, and for each pixel point in the current frame image, a vector (two-dimensional vector) may correspond to the vector matrix, representing the displacement direction and displacement magnitude of the pixel point relative to the historical frame image. For example, Figure 4 As shown, assuming that both the current frame image and the historical frame image include pixel points a, b, c, d, e, and f, the motion vector includes two-dimensional vectors corresponding to these pixel points respectively. In the two-dimensional vector, the x-coordinate represents the displacement of the pixel point on the x-axis, and the y-coordinate represents the displacement of the pixel point on the y-axis. If the displacement is positive, it means that the pixel point moves in the positive direction of the coordinate axis. If the displacement is negative, it means that the pixel point moves in the negative direction of the coordinate axis.

[0122] exist Figure 4 Based on the coordinate axes shown, assume that in the historical frame image, the coordinates of pixel a are (0, 0), the coordinates of pixel b are (3, 0), the coordinates of pixel c are (3, 1), the coordinates of pixel d are (1, 2), the coordinates of pixel e are (2, 3), and the coordinates of pixel f are (0, 3). In the current frame image, the coordinates of pixel a are (1, 0), the coordinates of pixel b are (3, 0), the coordinates of pixel c are (1, 1), the coordinates of pixel d are (2, 1), the coordinates of pixel e are (0, 2), and the coordinates of pixel f are (3, 3). In the motion vector, the vector corresponding to pixel a is (1, 0), the vector corresponding to pixel b is (0, 0), the vector corresponding to pixel c is (-2, 0), the vector corresponding to pixel d is (1, -1), the vector corresponding to pixel e is (-2, -1), and the vector corresponding to pixel f is (3, 0). Then it can be inferred that for pixel point a, it moves one coordinate point in the positive direction of the x-axis, and its y-axis position does not change. For pixel point b, its position on both the x-axis and the y-axis does not change, and so on.

[0123] based on Figure 4 For example, we can get the pixel coordinate relationship between the historical frame image and the current frame image as follows:

[0124] Frame N-1 (x, y) = Frame N (i, j)-MotionVector(i, j), where Frame N (i, j) is the pixel coordinate in the current frame image, MotionVector(i, j) is the motion vector corresponding to the pixel, Frame N-1 (x, y) is the coordinate of the pixel in the historical frame image.

[0125] Therefore, in this implementation, the process of the electronic device predicting the current frame image based on the historical frame image and the motion vector may include: the electronic device predicts a current frame image based on the pixel values ​​of the pixel points in the historical frame image based on the above-mentioned pixel coordinate relationship formula, that is, the electronic device can determine the position of the pixel point in the current frame image in the historical frame image based on the above-mentioned pixel coordinate relationship formula, and predict the pixel value of the pixel point in the current frame image based on the pixel value of the pixel point at that position in the historical frame image to obtain a predicted image.

[0126] After obtaining the predicted image, the electronic device can align and splice the predicted image and the current frame image actually obtained to obtain a spliced ​​image. In one implementation, the electronic device can align and splice the predicted image and the current frame image in the channel dimension. Exemplarily, when the sizes of the predicted image and the current frame image are both (H, W, C), C represents the number of channels, and the size of the obtained spliced ​​image is (H, W, 2C).

[0127] S103: weighting the stitched image based on the image depth information.

[0128] It can be understood that for a frame of image, the pixel changes at the edges of objects (such as people, objects, etc.) contained in the image or at positions where objects overlap are usually more obvious, and the depth value changes of the pixels at the corresponding positions are also large. Then the subsequent neural network model needs to focus on this part of the image when performing image super-resolution processing to optimize the image quality problems that may occur in this part. Therefore, the electronic device can assign weights to the spliced ​​images based on the image depth information, assigning larger weights to pixels with larger corresponding depth value changes in the spliced ​​images, and assigning smaller weights to pixels with smaller corresponding depth value changes in the spliced ​​images. Optionally, the electronic device can use an attention mechanism to assign weights to the spliced ​​images based on the image depth information.

[0129] In one achievable manner, since the image depth information obtained by the electronic device generally has no range regularity, the electronic device may first normalize the image depth information, for example, normalize it to the range of [0, 1]. Then, the normalized image depth information is multiplied by the vector corresponding to the spliced ​​image to obtain an image vector with weight distribution.

[0130] In another possible implementation, the image depth information may be a two-dimensional matrix, where each value corresponds to the depth value of the pixel at the corresponding position in the image. When the electronic device weights the stitched image, the normalized image depth information may be stitched with the stitched image again. For example, assuming that the size of the stitched image is (H, W, 2C), if the image depth information is stitched, the final data size is (H, W, 2C+1).

[0131] In actual scenarios, due to the memory limitation of electronic devices, if the stitched images are directly stored and subsequently operated, insufficient memory may occur. Therefore, in order to reasonably allocate memory, in one embodiment, the electronic device can rearrange the stitched images (space to depth), and then weight the stitched images after the rearrangement. It should be noted here that space to depth is to move the spatial data of the image (i.e., the width and height dimensions) to the depth dimension, which is equivalent to dividing the enlarged first image into multiple small images, and arranging the multiple small images in the order of depth according to their positions in the first image. Alternatively, in another embodiment, the electronic device can also weight the stitched images based on the image depth information, and then rearrange the stitched images after the weight allocation (space to depth).

[0132] S104, inputting the weighted stitched image into a neural network model for image super-resolution processing to obtain a high-resolution current frame image, and displaying the current frame image.

[0133] Among them, the neural network model used in this step is a trained network model, which can realize the conversion process from low-resolution images to high-resolution images, and the neural network model also focuses on processing the details of the objects contained in the image during training, for example, the features of the objects or characters contained in the image can be refined to make the picture quality of the obtained high-resolution image better. In addition, the neural network model can receive input of multiple frames of information, that is, the electronic device can input the spliced ​​image after the above weight distribution into the neural network model, and the neural network model performs image super-resolution processing based on the multiple frames of information in the spliced ​​image to obtain a high-resolution current frame image.

[0134] It should be noted that the above neural network model can be applicable to image inputs of different resolutions as long as the set input format is met. For example, the input format can be the above-mentioned (H, W, 2C+1), where H and W represent the height and width of the input image respectively, and C represents the number of channels, for example C=3.

[0135] In one embodiment, the neural network model may be a network model including N convolutional layers, for example, a network model including 7 convolutional layers, each of which has corresponding network parameters. Taking a network model with 7 convolutional layers as an example, the network parameters of the neural network model in the embodiment of the present application may be:

[0136] res=tf.keras.layers.Conv2D(filters=F1, kernel_size=K1, strides=S1, padding=P1)(res);

[0137] res=tf.keras.layers.Conv2D(filters=F2, kernel_size=K2, strides=S2, padding=P2)(res);

[0138] res=tf.keras.layers.Conv2D(filters=F3, kernel_size=K3, strides=S3, padding=P3)(res);

[0139] res=tf.keras.layers.Conv2D(filters=F4, kernel_size=K4, strides=S4, padding=P4)(res);

[0140] res=tf.keras.layers.Conv2D(filters=F5, kernel_size=K5, strides=S5, padding=P5)(res);

[0141] res=tf.keras.layers.Conv2D(filters=F6, kernel_size=K6, strides=S6, padding=P6)(res);

[0142] res=tf.keras.layers.Conv2D(filters=F7, kernel_size=K7, strides=S7, padding=P7)(res);

[0143] Among them, the above are the network parameters of 7 convolutional layers from top to bottom. Filters represents the number of filters of the convolutional layer, that is, the size of the output data of each convolutional layer; kernel_size represents the size of the convolution kernel in the convolutional layer, for example, it can be 3×3; strides represents the size relationship between the input data and output data of the convolutional layer. The larger the strides, the smaller the output data is relative to the input data. For example, when strides=1, when the input data is an image, the size of the output image and the output image remains unchanged. When strides is greater than 1, the size of the output image relative to the output image becomes smaller; padding represents the data padding method, which usually includes two padding methods: valid and same. The valid method means that there is no pad (padding), and the size of the output data may change after convolution. The same method means that the size of the output data and the input data after padding is the same.

[0144] After the electronic device inputs the stitched image after the weight distribution into the neural network model, it can obtain a high-resolution current frame image through convolution operations of N convolution layers.

[0145] From the above description, it can be seen that due to the memory limitation of the electronic device, the electronic device can rearrange the spliced ​​image (space to depth), and then assign weights to the spliced ​​image after the image rearrangement. Then, the spliced ​​image after the image rearrangement is input to the neural network model. Figure 5 As shown, the process of image super-resolution processing by an electronic device through a neural network model may include: the electronic device first predicts a predicted image of the current frame based on the historical frame image and the motion vector, and then aligns and splices the predicted image and the current frame image to obtain a spliced ​​image, and then rearranges the spliced ​​image to obtain multiple small images, and then weights the spliced ​​image after the image rearrangement (i.e., multiple small images). After that, the electronic device can input the weighted spliced ​​image into the neural network model for convolution operation. It can be understood that the input result of the neural network model should be the output result corresponding to the multiple small images, such as multiple vector matrices. If you want to get a result corresponding to the spliced ​​image, you need to perform the inverse operation of the above-mentioned image rearrangement (depth to space) on the output results corresponding to the multiple small images, that is, to obtain a high-resolution current frame image.

[0146] In this embodiment, image rearrangement can be achieved by res=tf.nn.space_to_depth(inp,M) function, where M represents the scale of image rearrangement, for example, space_to_depth(inp,3) represents image rearrangement at 3 scales. In this scenario, exemplarily, the network parameters of the network model with 7 convolutional layers can be:

[0147] res=tf.keras.layers.Conv2D(filters=32, kernel_size=3, strides=1, padding='same')(res);

[0148] res=tf.keras.layers.Conv2D(filters=32, kernel_size=3, strides=1, padding='same')(res);

[0149] res=tf.keras.layers.Conv2D(filters=32, kernel_size=3, strides=1, padding='same')(res);

[0150] res=tf.keras.layers.Conv2D(filters=32, kernel_size=3, strides=1, padding='same')(res);

[0151] res=tf.keras.layers.Conv2D(filters=32, kernel_size=3, strides=1, padding='same')(res);

[0152] res=tf.keras.layers.Conv2D(filters=32, kernel_size=3, strides=1, padding='same')(res);

[0153] res=tf.keras.layers.Conv2D(filters=27, kernel_size=3, strides=1, padding='same')(res);

[0154] In this example, if the image is rearranged at 3 scales, the number of small images obtained is 27. Therefore, in the last convolution layer, filters=27 should be set, that is, to ensure that the number of output results corresponding to the final small image is also 27. In this way, when the output result is subjected to the inverse operation of image rearrangement, the current frame image with the same resolution as the historical frame image can be obtained. It can be understood that the number of filters in the first to sixth convolution layers may not be limited to 32, but may be other numbers, such as 16, 64, etc.

[0155] Optionally, in one embodiment, the electronic device may move the above-mentioned image super-resolution processing process to the NPU for processing to reduce the GPU load.

[0156] It can be understood that after the electronic device obtains a high-resolution current frame image, it can be stored in a shared memory for use in processing the next frame image. The high-resolution current frame image can be used as a historical frame image to perform image super-resolution processing on the next frame image.

[0157] It can also be understood that when the electronic device acquires the first frame image, since the first frame image has no corresponding historical frame image, the electronic device can input the first frame image alone into the above-mentioned neural network model for image super-resolution processing to obtain a high-resolution first frame image, and the obtained high-resolution first frame image can be used as the historical frame image corresponding to the second frame image to perform image super-resolution processing on the second frame image, and so on.

[0158] In the above-mentioned image processing method, the electronic device can perform image super-resolution processing on the current frame image based on multiple frame information such as high-resolution historical frame images, current frame images, motion vectors between historical frame images and current frame images, and image depth information to obtain a high-resolution current frame image, and then display the high-resolution current frame rate image. After this process, the display image of the game application improves the resolution while also taking into account the continuity between the current frame image and the historical frame image, which can reduce problems such as flickering and ghosting of the game screen, making the picture quality in the game scene better.

[0159] In some scenarios, for the process of aligning and splicing the current frame image and the historical frame image according to the current frame image, the high-resolution historical frame image and the motion vector in the above S102 to obtain the spliced ​​image, that is, the electronic device first predicts a predicted image of the current frame according to the historical frame image and the motion vector, and then aligns and splices the predicted image and the current frame image to obtain the spliced ​​image. Because the resolution of the high-resolution historical frame image is greater than the resolution of the current frame image, the resolution of the predicted image predicted according to the historical frame image and the motion vector may also be greater than the resolution of the current frame image. Therefore, if the predicted image and the current frame image are to be aligned, the electronic device can perform an interpolation and amplification operation on the current frame image, and the enlarged current frame image has the same size as the predicted image, and then align the predicted image with the interpolated and amplified current frame image. It can be understood that at this time, the size of the motion vector obtained by the electronic device from the rendering engine is the same as the size of the enlarged current frame image and the predicted image.

[0160] In this embodiment, taking into account the fact that in actual applications, pixels in the image may be blocked, or the pixel values ​​of the same pixel may be inconsistent due to factors such as lighting, or pixel rendering may be lost during the image rendering process, etc. These abnormal situations may cause the pixel value errors in the predicted image or the current frame image after interpolation and amplification to be too large. Therefore, in an embodiment of the present application, the electronic device can perform error statistics on the predicted image and the current frame image after interpolation and amplification, and correct the pixel values ​​with excessive errors, and then perform splicing after correction.

[0161] In one achievable manner, the electronic device can calculate the value of the G(p)=B3×(F N-1 (p)-F N (p)) is used to calculate the pixel value error between the predicted image and each pixel in the current frame image after interpolation and amplification, where p is the position of the pixel, F N-1 is the predicted image, F N is the current frame image after interpolation and amplification, and B3 is a 3×3 filter (box filter). In addition, the electronic device can also set an error threshold W. If the calculated pixel value error corresponding to a certain pixel point is greater than W, it is determined that the pixel value of the pixel point needs to be corrected.

[0162] Alternatively, if Figure 6As shown, the process of the electronic device correcting the pixel value of pixel point A may include: first determining the position h of pixel point A in the current frame image initially acquired, and then finding the pixel values ​​of the pixels around position h. Assuming that there are three pixels B, C, and D around position h, the electronic device may then determine the correction value of pixel point A based on the pixel values ​​of the surrounding pixels. For example, the electronic device may determine the correction value of pixel point A based on the pixel values ​​of the surrounding pixels based on the clamp operation. For example Figure 6 As shown in , there is a line L between the pixel value A1 of pixel point A in the predicted image and the pixel value A2 in the current frame image after interpolation and amplification, there is a limited range O between the pixel values ​​of pixel points B, C, and D, and there is an intersection point K between L and O. Then the electronic device can use the pixel value corresponding to K as the correction value of pixel point A, and replace the correction value into the predicted image and the current frame image after interpolation and amplification to obtain the final spliced ​​image. In this embodiment, by performing error correction on the predicted image and the current frame image after interpolation and amplification, the accuracy of the obtained spliced ​​image can be improved, thereby improving the accuracy of the subsequent high-resolution current frame image.

[0163] In actual scenarios, for example, when the current frame image to be displayed is a screen to be displayed of a running game application, the current frame image usually contains text, such as text prompts in the game, player speeches, etc. If the current frame image with text is subjected to super-resolution processing, it may cause text distortion and other phenomena. Therefore, before processing the current frame image, the embodiment of the present application can first separate the text. Figure 7 FIG. 1 is a flow chart of another image processing method provided in an embodiment of the present application. The method may include:

[0164] S201, obtaining a current frame image to be displayed by an electronic device, high-resolution historical frame images, motion vectors, and image depth information.

[0165] S202, performing text recognition on the current frame image, if it is recognized that the current frame image contains text, then performing text separation on the current frame image to obtain separated text and the current frame image containing only the image.

[0166] Among them, the electronic device can use a text recognition algorithm to recognize the current frame image, and the text recognition algorithm can include but is not limited to a maximally stable extremal regions (MSER) ​​algorithm, a connectionist text proposal network (CTPN) algorithm, etc. When the electronic device recognizes that the current frame image contains text, the text can be separated, for example, by using a segmentation algorithm to segment the text to obtain the separated text and the current frame image containing only the image. Optionally, the segmentation algorithm can include but is not limited to an edge segmentation algorithm, a region segmentation algorithm, etc.

[0167] It is understandable that the historical frame image may also contain text, so when executing this step, the electronic device can also perform text recognition on the historical frame image at the same time to separate the text therein.

[0168] S203, obtaining a spliced ​​image according to the current frame image that only contains images, the high-resolution historical frame images, and the motion vector.

[0169] S204: weight the stitched image based on the image depth information.

[0170] S205, inputting the weighted stitched image into a neural network model for image super-resolution processing to obtain a high-resolution current frame image.

[0171] The processes from S203 to S205 refer to S102 to S104 in the above embodiment, which will not be described in detail here.

[0172] S206, splicing the high-resolution current frame image and the separated text, and displaying the spliced ​​image.

[0173] When separating the text, the electronic device can record the coordinate position of the text in the current frame image. After obtaining the high-resolution current frame image, the text can be spliced ​​on the high-resolution current frame image according to the coordinate position of the text, and then the spliced ​​image can be displayed. Because the text does not participate in the image processing process, the displayed text is still the original text, which does not affect the display effect of the text, and further improves the picture quality of the final displayed image.

[0174] For the neural network model used in the above image processing method, before the electronic device performs image super-resolution processing, the neural network model needs to be trained first, and after the converged neural network model is trained, it is moved to the electronic device for processing to improve the accuracy of the processing result. Therefore, in the following embodiments, the training process of the above neural network model will be described in detail. Figure 8 As shown, the training process may include:

[0175] S301, obtaining a training image set.

[0176] The training image set may include input image information and supervision images, the input image information is used to input the neural network model for processing, and the supervision image is used to supervise the difference between the output result of the neural network model and the supervision image, thereby supervising the learning and training process of the neural network model. It can be understood that one input image information can correspond to one supervision image, assuming that the resolution of the input image information is a and the resolution of the supervision image is b, then (a, b) is a set of input and output combinations; if the neural network model is used for actual image super-resolution processing later, if an image with a resolution of a is input, then an image with a resolution of b can be obtained.

[0177] In an embodiment of the present application, the input image information may include a low-resolution first image and a high-resolution second image, as well as a motion vector and image depth information between the two. The supervision image may include a high-resolution first image. The high-resolution image is used as a supervision signal to calculate the difference between the output image of the neural network model and the high-resolution image to train the neural network model. In the scenario of game applications, the training image set obtained by the electronic device may be a large number of images of different game screens, and the game image quality can usually be set in the game application, so the electronic device can obtain low-resolution images and high-resolution images by setting different game image qualities.

[0178] In this embodiment, in order to improve the processing capability of the neural network model for more image quality problems and to improve the accuracy of the output results of the neural network model, the electronic device may perform data enhancement preprocessing on the training image set, such as reducing the clarity of the input image (low-resolution image) so that it has more image quality problems, and improving the clarity of the supervision image (high-resolution image), to ensure that the neural network model can ultimately output more accurate results.

[0179] As a feasible method, for high-resolution images, the electronic device can use the GTX framework to change the rendering resolution to n times the original resolution, such as 1.5 times or 2 times, when capturing the image, to obtain a higher resolution image. Then, the electronic device uses an image super-resolution algorithm to optimize the image with higher resolution to further improve the image resolution. It can be seen from experiments that after processing with these two data enhancement methods, the edges of people and objects in the obtained high-resolution image are smoother and clearer. Optionally, the image super-resolution algorithm used by the electronic device may include but is not limited to the SRGAN (super-resolution of generative adversarial networks) algorithm.

[0180] As another feasible method, for low-resolution images, the electronic device can perform a downsampling strategy on them, and downsample the low-resolution images using a sampling coefficient of m. Optionally, the electronic device can perform multiple downsampling operations, such as using a nearest neighbor interpolation downsampling with a sampling coefficient of 0.8, a linear downsampling of 0.85, and a linear sampling of 1.0 to process the low-resolution image, to obtain an image with a lower resolution. It can be seen from experiments that through this three-downsampling strategy, more image quality problems can be added to the original low-resolution image, such as jagged edges of characters, blurred distant views, etc.

[0181] Exemplarily, the electronic device may implement the above three-time downsampling strategy in the following manner:

[0182] Assume that the resolution of the original low-resolution image is h×w=lr.size, the nearest neighbor interpolation downsampling process with a sampling factor of 0.8 is lr=hr.resize([h×0.8, w×0.8], 'nearest'), the linear downsampling process with a sampling factor of 0.85 is lr=hr.resize([h×0.85, w×0.85], 'linear'), and the linear sampling process with a sampling factor of 1.0 is lr=hr.resize([h, w], 'linear').

[0183] It can be understood that the downsampling strategy implemented by the electronic device is not limited to the above three downsamplings, and other sampling coefficients can also be used to downsample the low-resolution image, and the embodiments of the present application do not impose any restrictions on this.

[0184] As another feasible method, if the images in the training image set acquired by the electronic device contain text, the electronic device can also perform text separation processing on the images so that the images participating in the network model training process do not contain text, thereby avoiding the problem of blurred text in actual scenes.

[0185] S302, inputting the input image information into the neural network model to be trained to obtain an output image.

[0186] Among them, the neural network model to be trained, that is, the network model that has not been trained yet, may include an initial neural network model or a neural network model in training (both may be referred to as neural network models for short), but the process of inputting the input image information into the neural network model to be trained for processing is similar to the process of inputting the weighted assigned spliced ​​image into the neural network model for processing in the above S104, that is, it is also necessary to align and splice the first image and the second image in the input image information and assign weights, and then perform a convolution operation on the input image information through the N convolutional layers of the neural network model to obtain an output image.

[0187] It is understandable that since the neural network model has not been trained yet, the output image may not be of high quality. As the model training continues, the quality of the output image will become higher and higher.

[0188] S303, calculating the loss between the output image and the supervision image, and adjusting the network parameters of the neural network model according to the loss.

[0189] From the above description, it can be seen that the image quality of the output image is not high, that is, there is a certain error between the output image and the above-mentioned supervisory image (high-resolution third image). Therefore, the electronic device can calculate the error (loss) between the output image and the supervisory image, and reversely adjust the network parameters of the neural network model according to the loss. For example, starting from the last convolution layer of the neural network model, adjust the network parameters of each convolution layer in turn until the first convolution layer is adjusted. Optionally, the electronic device can use a loss function to calculate the loss between the output image and the supervisory image.

[0190] In one embodiment, the electronic device may use one or more loss functions to calculate the loss between the output image and the supervision image to supervise the edges, details, and multiple dimensions of objects such as characters, so as to ensure that the game screen can be improved in game scenes such as character edges and distant views. Exemplarily, the loss functions used in the embodiments of the present application may include:

[0191] (1) Regression Loss Function of Output Result

[0192] The regression loss function of the output result can characterize the error between the overall output image and the overall supervision image. From the above description, it can be seen that after the low-resolution input image is rearranged and input into the neural network model, the output results corresponding to multiple small images can be obtained, and the output results can be multiple vector matrices; and the supervision image can also be inversely converted into multiple vector matrices of the same number. The regression loss function of the output result is to calculate the error between the multiple vector matrices corresponding to the output image and the multiple vector matrices corresponding to the supervision image. Optionally, the electronic device can calculate the regression loss function value according to Regression Loss = tf.generator.losses().

[0193] (2) Output feature loss function Feature Loss

[0194] The output feature loss function can characterize the error between the features of the output image and the features of the supervisory image. Therefore, the electronic device needs to extract the features of the output image and the features of the supervisory image. Exemplarily, the electronic device can input the output image and the supervisory image into the feature extraction network for feature extraction, for example, the output image and the supervisory image can be input into the VGG19 network for feature extraction. Among them, the VGG19 network contains 19 hidden layers, each layer is used to extract features of different depths. In the embodiment of the present application, the shallow features extracted from the 5th layer of the VGG19 network are preferably used to calculate the error. That is to say, the electronic device inputs the output image into the VGG19 network to extract the features of the 5th layer, and inputs the supervisory image into the VGG19 network to extract the features of the 5th layer, and calculates the error between the two features. Optionally, the electronic device can calculate the output feature loss function value according to Feature Loss = MeanAbsoluteError (VGG195 (HR), VGG195 (LR)), where HR is the supervisory image and LR is the output image.

[0195] (3) Sobel Loss based on the Sobel operator

[0196] The edge loss function can characterize the error between the edge features of the person in the output image and the edge features of the person in the supervision image, so the electronic device needs to extract the edge features of the output image and the edge features in the supervision image. For example, the electronic device can use the Sobel operator to extract the edge features and then calculate the distance between the edge features.

[0197] Exemplarily, the process of extracting edge features using the Sobel operator may include:

[0198] input_gray=tf.image.rgb_to_grayscale(imgs);

[0199] Gx=tf.constant([[-1,0,1],[-2,0,2],[-1,0,1]], dtype='float32');

[0200] Gy=tf.constant([[-1,-2,-1],[0,0,0],[1,2,1]], dtype='float32');

[0201] Gx=tf.expand_dims(Gx, axis=2);

[0202] Gx=tf.expand_dims(Gx, axis=3);

[0203] Gy=tf.expand_dims(Gy, axis=2);

[0204] Gy=tf.expand_dims(Gy, axis=3);

[0205] x=tf.nn.conv2d(input_gray, Gx, strides=[1, 1, 1, 1], padding="SAME", name="hor");

[0206] y=tf.nn.conv2d(input_gray, Gy, strides=[1, 1, 1, 1], padding="SAME", name="vec");

[0207] G=tf.square(x)+tf.square(y);

[0208] The above process is to first determine the coordinates (x, y) of edge points such as people in the image, and then connect the edge points to obtain edge lines. For the supervised image (HR), the Sobel feature G can be obtained. HR , the output image (LR) can get the Sobel feature G LR The electronic device can then HR and G LR Calculate the edge loss function value. Optionally, the electronic device can calculate the edge loss function value according to Sobel Loss = Mean Absolute Error (G HR , G LR ) calculates the marginal loss function value.

[0209] (4) Pixel Loss Function

[0210] The pixel-level loss function can characterize the error between the pixel value of a pixel point in the output image and the pixel value of a pixel point in the supervision image, that is, the pixel value error of the pixel point at the corresponding position in the image for the output image and the supervision image. Optionally, the electronic device can calculate the pixel-level loss function value according to Pixel Loss = Mean Absolute Error (HR, LR).

[0211] For the above four loss functions, the embodiments of the present application may use any one or more of them to calculate the loss between the output image and the supervision image. If only one loss function is used to calculate the loss between the output image and the supervision image, the electronic device may adjust the network parameters of the neural network model only according to the loss obtained. If multiple loss functions are used to calculate the loss between the output image and the supervision image, the electronic device may weight the multiple losses calculated and adjust the network parameters of the neural network model according to the weighted losses, or the electronic device may adjust the network parameters of the neural network model in sequence according to the multiple losses.

[0212] It can be understood that the training process of the neural network model usually requires multiple rounds of training and adjustment of network parameters. Therefore, the electronic device can cyclically execute steps S302 to S303 according to the training image set until the training converges. Among them, the conditions for training convergence may include: the calculated loss between the output image and the supervision image is less than the loss threshold, that is, the output image and the supervision image are very close to the extent that the error can be ignored, and at this time, it can be characterized that the neural network model training converges.

[0213] It can also be understood that because the training process of the neural network model is computationally intensive and time-consuming, the training process can be executed on a server. After the server training is completed, the neural network model can be loaded into the GTX framework of the electronic device for the electronic device to subsequently perform the image processing process.

[0214] During the training process of the above-mentioned neural network model, the electronic device can use one or more loss functions to calculate the loss between the output image and the supervision image to supervise the edges, details, and multi-layer dimensions of characters, etc., thereby improving the processing accuracy of the neural network model and enabling higher quality images to be obtained during subsequent image super-resolution processing.

[0215] For the above image processing process, combined with the above Figure 2 The software structure framework shown in the figure shows that the electronic device can use the network model to perform image super-resolution processing on the current frame image obtained from the game application under the GTX framework. If combined with the QNN framework, the electronic device can Figure 5 The image stitching, weight distribution, image rearrangement (space to depth) algorithm and the inverse operation (depth to space) algorithm of image rearrangement are integrated into libsr.so, and the image super-resolution algorithm (i.e. the convolution algorithm of the neural network model) is integrated into libsr_skel.so for GTX framework to call.

[0216] During the execution of libsr.so, since it involves the process of rearranging the stitched image, that is, dividing the stitched image into multiple small images, it involves the storage of multiple small images. For this, the embodiment of the present application uses the CDSP-Software-based L2FETCH instruction set, which can load the data of multiple small images in blocks, that is, perform L2Prefetch rewrite on the HVX vector corresponding to the first image according to the size of the available space in the cache, specify the initial Prefetch address, width, stride and height, which greatly reduces the memory stall caused by CDSP in data loading. In addition, in the implementation process, in addition to using conventional calculation instructions, you can also use The provided binary pixel arrangement instructions and HVX 1024bit SIMD instructions improve the flexibility and processing efficiency of CDSP vector processing. It can be understood that the execution process of libsr.so can be executed on the CPU.

[0217] During the execution of libsr_skel.so, the image super-resolution algorithm is mainly involved. From the above description, it can be seen that the execution process can be transferred to the NPU to reduce the GPU load.

[0218] Based on this, combined with the above Figure 2 The software structure framework shown in the embodiment of the present application also provides a flowchart of an image processing method, which mainly involves the signaling interaction between various modules, such as Fig. 9 As shown, it may specifically include:

[0219] S11, after the game application is started, the model loading module initializes the neural network model.

[0220] The initialization of the neural network model includes but is not limited to model loading, model compilation, operation checking, and memory configuration.

[0221] S12, the rendering processing module selects a rendering method and changes the window resolution.

[0222] Here, the rendering processing module can select a rendering method according to the needs of the game application, such as forward rendering or delayed rendering. For example, in delayed rendering, after obtaining a high-resolution image, it can be stored in a shared memory before rendering. While executing this process, the electronic device can also allocate a shared memory for subsequent use.

[0223] In addition, since the embodiment of the present application aims to increase the low-resolution current frame image to a high-resolution one, the window resolution needs to be adapted to the high-resolution during subsequent rendering. Therefore, the rendering processing module can also change the window resolution.

[0224] S13, the data capture module obtains the current frame image to be displayed, the high-resolution historical frame images, the motion vector and the image depth information.

[0225] Among them, the data capture module can capture the current frame image, motion vector and image depth information to be displayed from the game application through the capabilities of the GTX framework, and can also obtain high-resolution historical frame images from the shared memory.

[0226] Optionally, after the data capture module captures the current frame image, it may also perform format conversion on it, for example, converting the RGB format into the YCbCr format, to facilitate subsequent processing.

[0227] S14, the data capture module sends the current frame image, high-resolution historical frame images and motion vectors to the stitching module.

[0228] S15, the stitching module obtains a stitched image according to the current frame image, the high-resolution historical frame images and the motion vector.

[0229] The alignment and splicing process in S15 can refer to S102 in the above embodiment, which will not be described in detail here. It can be understood that the splicing module can call the above libsr.so to perform the splicing process.

[0230] S16, the stitching module sends the stitched image to the operator optimization module.

[0231] S17, the operator optimization module rearranges the stitched image to obtain multiple small images.

[0232] Here, before using the neural network model to perform image super-resolution processing on the image to be displayed, the operator optimization module can call the above libsr.so to rearrange the spliced ​​image (space to depth) to optimize the memory usage problem. After obtaining multiple small images after image rearrangement, the subsequent weight distribution processing and image super-resolution processing can be performed.

[0233] S18, the operator optimization module sends multiple small images to the attention module.

[0234] S19, the data capture module sends the image depth information to the attention module.

[0235] S20, the attention module assigns weights to multiple small images based on image depth information.

[0236] The weight allocation process in S20 can refer to S103 in the above embodiment, which will not be described in detail here. It can be understood that the attention module can call the above libsr.so to perform the weight allocation process.

[0237] S21, the attention module sends multiple small images with weight assignment to the model operation module.

[0238] S22, the model operation module calls libsr_skel.so, and uses a neural network model to perform image super-resolution processing on the multiple small images after weight allocation, and obtains output results corresponding to the multiple small images.

[0239] Here, the model operation module can call libsr_skel.so to perform the image super-resolution processing process on the NPU. The process can refer to the description of S105 above, which will not be repeated here. The output results corresponding to the multiple small images can be multiple vector matrices.

[0240] S23, the model operation module sends the output results corresponding to the multiple small images to the operator optimization module.

[0241] S24, the operator optimization module performs an inverse operation of image rearrangement on the output results corresponding to the multiple small images to obtain a current frame image with a high resolution.

[0242] After obtaining the output results corresponding to multiple small images, if you want to obtain the result corresponding to the current frame image, you need to perform the inverse operation of the above-mentioned image rearrangement (depth to space) on the output results corresponding to the multiple small images respectively, that is, to obtain a high-resolution current frame image.

[0243] S25, the operator optimization module sends the high-resolution current frame image to the image output module.

[0244] S26, the image output module displays the current frame image with high resolution.

[0245] The image output module may call a display driver to display the high-resolution current frame image. Optionally, if the RGB format of the current frame image is converted to the YCbCr format, the image output module here also needs to convert the high-resolution current frame image in the YCbCr format to the RGB format, and then render and display it.

[0246] The above-mentioned image processing method performs image super-resolution processing on the current frame image based on multiple frame information such as high-resolution historical frame images, current frame images, motion vectors between historical frame images and current frame images, and image depth information to obtain a high-resolution current frame image, and then display the high-resolution current frame rate image. After this process, the display image of the game application improves the resolution while also taking into account the continuity between the current frame image and the historical frame image, which can reduce problems such as flickering and ghosting of the game screen, making the picture quality in the game scene better. In addition, the super-resolution processing process performed by the neural network model in the embodiment of the present application can be transferred to the NPU for processing, reducing the load of the GPU, that is, reducing the processing power consumption of the electronic device.

[0247] Based on the image processing method provided in the embodiment of the present application, we also conducted corresponding experimental comparisons, comparing the operating effects of not adopting the image processing method provided in the embodiment of the present application (scenario 1) and adopting the image processing method provided in the embodiment of the present application (scenario 2) in the game scenario, where the image processing method provided in the embodiment of the present application is not adopted, that is, the original low-resolution image is used for display. Fig.10 This is the display effect of the game screen when scene 1 and scene 2 are running. Fig.10 It can be seen that when the image processing method provided in the embodiment of the present application is adopted (ie, scenario 2), the display quality is significantly improved, especially the edge processing of characters is smoother, and problems such as screen flickering are also reduced.

[0248] The following table 1 is a comparison table of the power consumption of electronic devices when running scenarios 1 and 2. Figure 1 It can be seen that when the image processing method provided in the embodiment of the present application is adopted (ie, scenario 2), the power consumption of the electronic device does not change much, it can run smoothly on a mobile electronic device, and the jitter rate and the number of frame losses are reduced.

[0249] Table 1

[0250] Test Application Scenario Full Frame Average frame rate Low frame rate Jitter rate Maximum number of dropped frames Application A Scenario 2 60 51.93 2.5％ 4.35％ 22 Application A Scenario 1 60 55.89 0.75％ 5.96％ 23 Test Application Scenario Power consumption (mA) Median frame rate CPU-C1 Load CPU-C2 Load CPU-C3 Load Application A Scenario 2 1678.47 52 44 65.23 81.65 Application A Scenario 1 1618.7 56 43.64 71.16 84.43

[0251] The above describes in detail an example of an image processing method provided by an embodiment of the present application. It is understandable that, in order to implement the above functions, the electronic device includes hardware and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to exceed the scope of the present application.

[0252] The embodiment of the present application can divide the functional modules of the electronic device according to the above method example. For example, each function can be divided into various functional modules, such as a detection unit, a processing unit, a display unit, etc., or two or more functions can be integrated into one module. The above integrated module can be implemented in the form of hardware or in the form of a software functional module. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0253] It should be noted that all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0254] The electronic device provided in this embodiment is used to execute the above-mentioned image processing method, and thus can achieve the same effect as the above-mentioned implementation method.

[0255] In the case of an integrated unit, the electronic device may further include a processing module, a storage module and a communication module. The processing module may be used to control and manage the actions of the electronic device. The storage module may be used to support the electronic device to execute stored program codes and data, etc. The communication module may be used to support the electronic device to communicate with other devices.

[0256] Among them, the processing module can be a processor or a controller. It can implement or execute various exemplary logic boxes, modules and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a digital signal processor (DSP) and a microprocessor, and so on. The storage module can be a memory. The communication module can specifically be a device that interacts with other electronic devices, such as a radio frequency circuit, a Bluetooth chip, a Wi-Fi chip, etc.

[0257] In one embodiment, when the processing module is a processor and the storage module is a memory, the electronic device involved in this embodiment may be a Figure 1 Device of the structure shown.

[0258] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the processor executes the image processing method of any of the above embodiments.

[0259] The embodiment of the present application also provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute the above-mentioned related steps to implement the image processing method in the above-mentioned embodiment.

[0260] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor can execute the computer-executable instructions stored in the memory so that the chip executes the image processing method in the above-mentioned method embodiments.

[0261] Among them, the electronic device, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above and will not be repeated here.

[0262] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0263] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0264] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0265] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0266] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0267] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An image processing method, the method being executed by an electronic device, characterized in that: The method comprises: Acquire a current frame image to be displayed by the electronic device, a historical frame image with a first resolution, a first vector, and image depth information corresponding to the current frame image, wherein the resolution of the historical frame image is greater than the resolution of the current frame image, the first vector represents the displacement between the position of the same pixel point in the current frame image and the position in the historical frame image, the current frame image is an image of a game application running on the electronic device, and the first vector and the image depth information are information acquired from a rendering engine of the game application; Based on a preset pixel coordinate relationship, a predicted image for the current frame image is determined according to the coordinates of each pixel point in the historical frame image, the pixel value and the first vector, wherein the preset pixel coordinate relationship includes the corresponding relationship between the coordinates of each pixel point in the historical frame image, the coordinates of each pixel point in the predicted image and the first vector; Aligning and splicing the predicted image and the current frame image to obtain a spliced ​​image; Inputting the stitched image and the image depth information into a neural network model to perform image super-resolution processing to obtain a current frame image having the first resolution, and displaying the current frame image having the first resolution; Before inputting the stitched image and the image depth information into a neural network model for image super-resolution processing, the method further includes: Acquire a training image set, the training image set including input image information and a supervisory image, the input image information including a first image at a second resolution, a second image at the first resolution, a second vector, and image depth information corresponding to the first image, the second vector representing a displacement between a position of a same pixel point in the first image and a position in the second image, the supervisory image including the first image at a first resolution, the first resolution being greater than the second resolution; Inputting the input image information into the neural network model to be trained to obtain an output image; Calculating the loss between the output image and the supervision image, and adjusting the network parameters of the neural network model to be trained according to the loss to obtain the neural network model through training; The loss includes at least one of a first loss, a second loss, a third loss and a fourth loss, wherein the first loss represents an error between multiple vector matrices corresponding to the output image and multiple vector matrices corresponding to the supervisory image, the second loss represents an error between features of the output image and features of the supervisory image, the third loss represents an error between edge features of an object contained in the output image and edge features of an object contained in the supervisory image, and the fourth loss represents an error between pixel values ​​of pixels in the output image and pixel values ​​of pixels in the supervisory image.

2. The method according to claim 1, characterized in that Before aligning and splicing the predicted image and the current frame image, the method further includes: Performing an interpolation and enlargement operation on the current frame image to obtain an enlarged current frame image, wherein the size of the enlarged current frame image is the same as the size of the predicted image; Correspondingly, aligning and splicing the predicted image and the current frame image to obtain the spliced ​​image includes: The predicted image and the enlarged current frame image are aligned and spliced ​​to obtain the spliced ​​image.

3. The method according to claim 2, characterized in that After obtaining the enlarged current frame image, the method further includes: Calculating the pixel value error between each pixel point in the predicted image and the enlarged current frame image; If the pixel value error corresponding to the first pixel point is greater than the error threshold, the pixel value of the first pixel point in the predicted image and / or the pixel value of the first pixel point in the enlarged current frame image is corrected, and the first pixel point is any pixel point in the predicted image and the enlarged current frame image.

4. The method according to any one of claims 1 to 3, characterized in that The step of inputting the stitched image and the image depth information into the neural network model for image super-resolution processing to obtain a current frame image having the first resolution includes: Performing weight allocation on the stitched image based on the image depth information; The weighted spliced ​​image is input into the neural network model for image super-resolution processing to obtain a current frame image of the first resolution.

5. The method according to claim 4, characterized in that After obtaining the stitched image, the method further includes: Rearranging the stitched image to obtain a plurality of small images arranged according to image depth; Accordingly, the weight allocation for the stitched image based on the image depth information includes: Based on the image depth information, weights are allocated to the multiple small images.

6. The method according to any one of claims 1 to 3, characterized in that After acquiring the training image set, the method further includes: Performing image super-resolution processing on the supervision image to obtain a supervision image with improved resolution; The first image with the second resolution is processed by reducing the resolution to obtain the first image with reduced resolution.

7. The method according to any one of claims 1 to 3, characterized in that After acquiring the current frame image to be displayed by the electronic device, the method further includes: Performing text recognition on the current frame image; If it is recognized that the current frame image contains text, the text is separated from the current frame image to obtain the separated text and the current frame image containing only the image.

8. The method according to claim 7, characterized in that After inputting the stitched image and the image depth information into a neural network model for image super-resolution processing to obtain a current frame image with the first resolution, the method further includes: The current frame image of the first resolution and the separated text are spliced ​​to obtain a current frame image containing text.

9. The method according to any one of claims 1 to 3, characterized in that The electronic device includes an embedded neural network processor NPU, and the step of inputting the spliced ​​image and the image depth information into a neural network model for image super-resolution processing includes: The stitched image and the image depth information are input into the neural network model through the NPU to perform image super-resolution processing.

10. The method according to any one of claims 1 to 3, characterized in that The neural network model is a network model including 7 convolutional layers.

11. An electronic device, characterized in that: include: one or more processors; one or more memories; The memory stores one or more programs, and when the one or more programs are executed by the processor, the electronic device executes the method according to any one of claims 1 to 10.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the processor calls instructions so that the electronic device executes the method according to any one of claims 1 to 10.

Images