Image processing method and device

Abstract

The embodiment of the invention provides an image processing method and device, and relates to the technical field of terminals, and the method comprises the steps that terminal equipment obtains a first image in a first application, the first image comprises an image located in a sniper lens and an image located outside the sniper lens, and the number of pixel points of the image in the sniper lens is a first numerical value; the terminal equipment inputs the image in the sniper lens into a preset model and outputs a second image, the preset model is used for carrying out AI super-resolution processing on the image in the sniper lens, the number of pixel points of the second image is a second numerical value, and the second numerical value is larger than the first numerical value; the terminal equipment splices the second image and the image outside the sniper lens into a third image; and the terminal device displays the third image. Therefore, the terminal equipment can perform AI super-resolution processing on the image in the sniper lens in the first application, and does not perform AI super-resolution processing on the area outside the sniper lens, so that the definition of the image in the sniper lens is improved, and meanwhile, the effect of reducing the power consumption of the equipment is realized.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to an image processing method and device. Background Art

[0002] With the development of gaming technology in terminal devices, both personal computer (PC) gaming technology and mobile gaming technology have been upgraded. The emergence of innovative technologies such as hardware thread director and deep learning super sampling (DLSS) also enables the various units in the terminal device to work together to calculate, which not only enhances the computing power of the device, but also improves the user experience when playing games with the terminal device. With the increasing demand for device performance in mobile games, especially with the trend of thin and light mobile phone design, problems such as heat and power consumption have begun to become a shackle that affects the player's gaming experience. Therefore, in order to solve the problems of heat, power consumption, and battery life caused by high power consumption, mobile game super-resolution technology (Game AI-SR, or AI super-resolution technology for short) can render a lower resolution picture through the graphics processing unit (GPU), and then "brain fill" the picture close to the original resolution through the accelerated processing unit (APU) and algorithm, while restoring the details of the picture, reducing GPU power consumption.

[0003] Generally, AI super-resolution technology can be applied to shooting game applications. For example, when a terminal device receives a signal from a user that a sniper scope is turned on in a shooting game application, the terminal device can process the entire game scene using AI super-resolution technology, allowing the user to play the game in high-resolution images.

[0004] However, the above-mentioned image processing method consumes high power. Summary of the invention

[0005] The embodiments of the present application provide an image processing method and apparatus, so that a terminal device can perform AI super-resolution processing on images in a first application and within a sniper scope, while areas outside the sniper scope are not subjected to AI super-resolution processing, thereby improving the clarity of the image within the sniper scope and achieving the effect of reducing the power consumption of the device.

[0006] In the first aspect, the embodiment of the present application provides an image processing method, the method comprising: a terminal device receives a first operation for a first application, in response to the first operation, the terminal device displays a first interface, the first interface includes: a first button for opening a sniper scope; the terminal device receives a second operation for the first button; in response to the second operation, the terminal device obtains a first image in the first application, the first image includes an image located in the sniper scope and an image located outside the sniper scope, the number of pixels of the image in the sniper scope is a first value; the terminal device inputs the image in the sniper scope into a preset model, outputs a second image, the preset model is used to perform AI super-resolution processing on the image in the sniper scope, the number of pixels of the second image is a second value, and the second value is greater than the first value; the terminal device splices the second image and the image outside the sniper scope into a third image; the terminal device displays the third image. In this way, the terminal device can perform AI super-resolution processing on the images in the first application and in the sniper scope, and the area outside the sniper scope is not subjected to AI super-resolution processing, thereby improving the clarity of the image in the sniper scope and achieving the effect of reducing the power consumption of the device.

[0007] The first application may be a shooting game application described in the embodiment of the present application, and the first interface may be Figure 1 In the interface shown in a in FIG. 1 , the first button may be button 201, and the first image may be Figure 1 The scene image in the interface shown in b, the image inside the sniper scope may be the image inside the zoom scope described in the embodiment of the present application, the image outside the sniper scope may be the image outside the zoom scope described in the embodiment of the present application, the second image may be the image inside the zoom scope that has been processed by AI super-resolution as described in the embodiment of the present application, and the third image may be the target image described in the embodiment of the present application.

[0008] In a possible implementation, a correspondence between the magnification of the sniper scope and the pixel increase multiple is preset in the terminal device, and the pixel increase multiple is the ratio of the second value to the first value; the terminal device inputs the image in the sniper scope into the preset model and outputs the second image, including: when the magnification of the sniper scope is a third value, the terminal device obtains a fourth value corresponding to the third value from the correspondence; the correspondence includes a correspondence between the third value and the fourth value, the terminal device inputs the fourth value and the image in the sniper scope into the preset model, outputs the second image, and the ratio of the second value to the first value is the fourth value. In this way, the terminal device can adjust the resolution of the content displayed in the sniper scope more accurately according to the magnification of the sniper scope to improve the clarity of the picture.

[0009] In a possible implementation, in response to the second operation, the terminal device obtains the first image in the first application, including: in response to the second operation, the terminal device generates a first identifier corresponding to the first operation, the first identifier being an identifier corresponding to opening the sniper scope; when the terminal device determines that the first identifier meets a preset identifier, the terminal device obtains the first image in the first application. In this way, the terminal device can filter out the scene in which the user opens the scope through scene recognition, so that the terminal device can subsequently perform AI super-resolution processing on the image in the scope.

[0010] In a possible implementation, after the terminal device acquires the first image in the first application, the method further includes: the terminal device identifies at least one circle in the first image through image feature recognition; the terminal device determines the largest circle among the at least one circle, and the content displayed in the largest circle is the image inside the sniper scope; the terminal device divides the first image into the image inside the sniper scope and the image outside the sniper scope. In this way, the terminal device can accurately identify the image inside the sniper scope through the circle recognition in the image feature recognition, and obtain the image inside the sniper scope and the image outside the sniper scope through image segmentation.

[0011] In a possible implementation, the terminal device includes an NPU, and the terminal device inputs the image in the sniper scope into a preset model and outputs a second image, including: the terminal device uses the NPU to input the image in the sniper scope into the preset model and outputs the second image. In this way, the terminal device can perform AI super-resolution processing on the image in the sniper scope in the NPU, which relieves the GPU computing pressure and reduces the power consumption of the device.

[0012] In a possible implementation, before the terminal device receives the first operation for the first application, the method further includes: the terminal device receives a third operation for the first application, the third operation is used to start the first application; in response to the third operation, the terminal device loads a preset model. To facilitate the subsequent use of the preset model, the terminal device can load the preset model when starting the first application.

[0013] In a possible implementation, TGPA is integrated in the first application, and the terminal device includes: a game optimization module and a game service processing module, and the terminal device generates a first identifier corresponding to the first operation, including: the first application generates a first identifier corresponding to the first operation through TGPA, and sends the first identifier to the game optimization module; the game optimization module sends the first identifier to the game service processing module; the terminal device determines that the first identifier meets the preset identifier, including: the game service processing module determines that the first identifier meets the preset identifier. Since TGPA can connect the terminal device and the shooting game application, and build a bridge between the shooting game application and the device manufacturer, the first application can communicate with the local module of the terminal device through the integrated TGPA to achieve the issuance of the first identifier.

[0014] In a possible implementation, the terminal device further includes: a graphics acceleration engine, and the method further includes: the graphics acceleration engine divides the first image into an image inside the sniper scope and an image outside the sniper scope; the terminal device splices the second image and the image outside the sniper scope into a third image, including: the graphics acceleration engine splices the second image and the image outside the sniper scope into the third image. The graphics acceleration engine can realize the splicing of images.

[0015] In a possible implementation, the terminal device further includes: a display synthesis system, a hardware synthesis processor, a liquid crystal display driver, and a display, and the method further includes: the display synthesis system synthesizes the third image with at least one layer to obtain a fourth image; the display synthesis system sends the fourth image to the display via the hardware synthesis processor and the liquid crystal display driver; the display displays the fourth image. Since the display synthesis system can have a layer synthesis capability, the third image can be synthesized with at least one layer in the terminal device to enrich the screen content.

[0016] In a possible implementation, the first application is a shooting game application, and the sniper scope includes one or more of the following: a 2x scope, a 3x scope, a 4x scope, a 6x scope, an 8x scope, or a tactical scope. Since a multi-fold scope usually needs to magnify the image in the scope, the image processing method for the multi-fold scope enables the terminal device to display a clearer picture in the first application to improve the user's gaming experience.

[0017] In a second aspect, an embodiment of the present application provides an image processing device, including a processing unit for receiving a first operation for a first application, and in response to the first operation, a display unit for displaying a first interface, wherein the first interface includes: a first button for opening a sniper scope; a processing unit for receiving a second operation for the first button; in response to the second operation, the processing unit is further used to obtain a first image in the first application, wherein the first image includes an image located within the sniper scope and an image located outside the sniper scope, and the number of pixels of the image within the sniper scope is a first value; the processing unit is further used to input the image within the sniper scope into a preset model and output a second image, wherein the preset model is used to perform AI super-resolution processing on the image within the sniper scope, and the number of pixels of the second image is a second value, which is greater than the first value; the processing unit is further used to splice the second image and the image outside the sniper scope into a third image; and the display unit is further used to display the third image.

[0018] In a third aspect, an embodiment of the present application provides a terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the terminal device executes the method described in any one of the implementations of the first aspect.

[0019] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores instructions. When the instructions are executed, the computer executes the method described in any implementation of the first aspect.

[0020] In a fifth aspect, a computer program product includes a computer program, which, when executed, enables a computer to execute the method described in any one of the implementations of the first aspect.

[0021] It can be understood that the effects corresponding to the second aspect to the fifth aspect can be found in the description of the first aspect and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A schematic diagram comparing an AI super-resolution technology and a traditional image processing method provided in an embodiment of the present application;

[0023] Figure 2 A schematic diagram of a scenario provided for an embodiment of the present application;

[0024] Figure 3 A schematic diagram of the hardware structure of a terminal device provided in an embodiment of the present application;

[0025] Figure 4 A schematic diagram of the software structure of a terminal device provided in an embodiment of the present application;

[0026] Figure 5A flowchart of an image processing method provided in an embodiment of the present application;

[0027] Figure 6 A schematic diagram of an interface for opening a sniper scope provided in an embodiment of the present application;

[0028] Figure 7 A schematic diagram of another interface for opening a sniper scope provided in an embodiment of the present application;

[0029] Figure 8 A schematic diagram of an AI super-resolution process provided in an embodiment of the present application;

[0030] Fig. 9 A schematic diagram of module interaction of an image processing method provided in an embodiment of the present application;

[0031] Fig.10 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0032] Fig.11 A schematic diagram of the hardware structure of another terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0033] In order to clearly describe the technical solution of the embodiment of the present application, the terms involved in the embodiment of the present application are introduced:

[0034] 1. AI super-resolution technology

[0035] The principle of AI super-resolution technology is to first render a lower-resolution image through the GPU, and then use the accelerated processing unit (APU) and algorithm to "fill in the blanks" with an image close to the original resolution to restore the image details.

[0036] For example, Figure 1 A schematic diagram comparing an AI super-resolution technology provided in an embodiment of the present application with a traditional image processing method.

[0037] like Figure 1 As shown, in the traditional rendering process, the terminal device renders a high-resolution image in the GPU, and then synthesizes the high-resolution image and other layers.

[0038] In the process of image rendering based on AI super-resolution technology, the terminal device renders low-resolution images in the GPU, and uses AI super-resolution technology in the neural process unit (NPU) to render high-resolution images, and then synthesizes the high-resolution images and other layers. Among them, AI super-resolution technology can be implemented based on the deep learning theory framework SNPE (snapdragon neural processing engine) framework or quantum neural networks (quantum neural networks, QNN) framework.

[0039] It is understandable that AI super-resolution alleviates the computing pressure of the GPU compared to the traditional rendering process. After the GPU rendering is completed, the model on the NPU is used for calculation to achieve the super-resolution effect and obtain the final high-resolution image. Compared with putting the entire image rendering process on the GPU for calculation, it greatly reduces the GPU load.

[0040] There are two directions for the application of AI super-resolution technology: one is to reduce chip power consumption while ensuring that the image quality is close to the original one. With the help of the "subtract first and then add" super-resolution technology, even if gamers choose high-definition mode to play games, the gaming experience will not be affected by heat and battery life issues; the second is to provide users with higher resolution and improve the user's immersive gaming experience. The image processing method described in the embodiment of this application is an improvement on the second point.

[0041] 2. Image segmentation technology

[0042] Image segmentation technology is a technology that divides an image into regions with different characteristics and extracts the target of interest. Image segmentation is an important part of digital image analysis. Among them, image threshold segmentation is a traditional and most commonly used image segmentation method. Because of its simple implementation, small amount of calculation and relatively stable performance, it has become the most basic and widely used segmentation technology in image segmentation. It is particularly suitable for images where the target and background occupy different grayscale ranges. In many cases, it is a necessary image preprocessing process before image analysis, feature extraction and pattern recognition.

[0043] The purpose of image thresholding is to divide the pixel set according to the grayscale, and each subset obtained forms a region corresponding to the real scene. Each region has consistent attributes, while the adjacent regions do not have such consistent attributes. Such division can be achieved by selecting one or more thresholds based on the grayscale. The basic principle of threshold segmentation is to divide the image pixels into several categories by setting different feature thresholds. Commonly used features include grayscale or color features directly from the original image, and features obtained by transforming the original grayscale or color values.

[0044] Image segmentation techniques may include: color image segmentation, grabCut algorithm segmentation, floodFill segmentation, watershed segmentation and other methods. The specific image segmentation method used in the embodiments of the present application is not limited.

[0045] Color image segmentation: Grayscale images are mostly segmented by operators to find edges and region growing fusion. Color images have added color information, and images can be segmented by different color values. Common color spaces HSV (hue, saturation, value) / HIS (hue, intensity, saturation), RGB (red, green, blue), LAB, etc. can all be used for segmentation. The inRange function can be used to implement thresholding. The inRange function provides a means of object detection. It uses a method based on the pixel value range to detect objects in the HSV color space to achieve the effect of segmentation.

[0046] GrabCut algorithm segmentation: The grabCut algorithm in OpenCV is an improvement of the Graph_Cut algorithm. Graph_Cut is an image segmentation technology directly based on the graph cut algorithm. It can confirm the foreground and background inputs to complete the optimal segmentation of the foreground and background. This algorithm uses the texture (color) information and boundary (contrast) information in the image. Only a small amount of user interaction is required to obtain a relatively good segmentation result. It is similar to the watershed algorithm, but the calculation speed is slower and the results are more accurate. If you want to extract foreground objects from static images (such as cutting objects from one image to another), the grabCut algorithm is the best choice.

[0047] FloodFill segmentation: Flood filling is a method of filling connected areas with a specific color, and achieving different filling effects by setting the upper and lower limits of connected pixels and the connection mode. Flood filling is often used to mark or separate a part of an image for further processing or analysis. It can also be used to obtain a mask area from an input image. The mask will speed up the processing process, or only process the pixels specified by the mask. The result of the operation is always a continuous area. The principle of flood filling is very simple, that is, starting from a point, traversing the nearby pixels, filling them with a new color until all pixels in the closed area are filled with the new color. Common implementation methods of floodFill filling include 4-neighborhood pixel filling method, 8-neighborhood pixel filling method, scan line-based pixel filling method, etc. In OpenCV, the flood filling algorithm is implemented by the floodFill function, which fills a connected domain from the seed point with the color we specify, and the connectivity is measured by the proximity of the pixel values.

[0048] Watershed segmentation: Watershed segmentation is an image region segmentation method. In the segmentation process, the similarity between adjacent pixels is used as an important reference, so that pixels with similar spatial positions and similar grayscale values ​​are connected to each other to form a closed contour. Closure is an important feature of the watershed algorithm. In OpenCV, the function of the watershed algorithm is watershed.

[0049] 3. Image stitching technology

[0050] Image stitching technology is used to stitch multiple images into one image. The image stitching method may include one or more of the following steps, for example: image matching, reprojection, stitching, or fusion.

[0051] Image matching refers to the geometric correspondence between a pair of images that describe the same scene.

[0052] Reprojection refers to converting a series of images into a common coordinate system through geometric transformation of the images.

[0053] Stitching refers to the process of creating a larger canvas image by merging the pixel values ​​of overlapping parts and keeping the pixel values ​​of non-overlapping parts.

[0054] Fusion refers to the discontinuity of objects usually caused by geometric and photometric migration errors, and produces visible seams near the boundary between two images. Therefore, in order to reduce the appearance of seams, fusion is required during or after stitching.

[0055] Image stitching methods can be divided into those based on "spatial domain" and those based on "frequency domain". Image stitching based on spatial domain can be further divided into region-based image stitching and feature-based image stitching. Feature-based image stitching can be further divided into low level feature-based image stitching and contour-based image stitching. Stitching based on low level features can be divided into four categories: stitching based on Harris corner detector, stitching based on feature point detection operator (features from accelerated segment testm, FAST) detector, stitching based on scale-invariant feature transform (SIFT) feature detector, and stitching based on speeded up robust features (SURF) feature detector.

[0056] According to the fusion method, stitching algorithms can be divided into transition smoothening-based and optimal seam-based. Smooth transition stitching can be further divided into feathering-based, pyramid-based, and gradient-based stitching.

[0057] In order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, words such as "first" and "second" are used to distinguish between identical or similar items with substantially the same functions and effects. For example, the first value and the second value are only used to distinguish different values, and their order is not limited. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit them to be different.

[0058] It should be noted that, in this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in this application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0059] In the present application, "at least one" means one or more, and "plurality" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, a and b, a and c, b and c, or a, b and c, where a, b, c can be single or multiple.

[0060] For example, Figure 2 A schematic diagram of a scenario provided in an embodiment of the present application. Figure 2 In the corresponding embodiments, a mobile phone as the terminal device is used as an example for illustration, and this example does not constitute a limitation on the embodiments of the present application.

[0061] When the terminal device receives a message that the user starts a shooting game application and enters the game scene, the terminal device can display the following information: Figure 2The interface shown in a in the figure may include: game character 200, button 201, button 202, button 203, target 204, target 205, a small map, and information such as the position mark at the top of the display screen. Among them, button 201 is used to open the sniper scope in any firearm to achieve aiming, button 202 is used to move the position of the game character, and button 203 is used to shoot. The game scene may include other targets besides target 204 and target 205, which is not limited in the embodiment of the present application.

[0062] like Figure 2 In the interface shown in a of FIG. 1 , when the user moves to a position where the target 204 can be aimed at by using the button 202, when the terminal device receives the user's trigger operation on the button 201, the terminal device can open the sniper scope in the firearm and display the following Figure 2 The interface shown in b. Figure 2 The interface shown in b may include: a sniper scope 206, and the sniper scope 206 may include a target 207. The target 207 may be a target obtained by magnifying the target 204 by the terminal device using the sniper scope 206.

[0063] Among them, the sniper scope 206 can be a multi-power scope, which is used to magnify the scene within the lens. Therefore, the sniper scope 206 can be at least one of the following, for example: a two-fold scope, a three-fold scope, a four-fold scope, a six-fold scope, an eight-fold scope and a tactical scope, etc. The tactical scope can realize the function of any sniper scope between a four-fold scope and an eight-fold scope, and the tactical scope can adjust the magnification of the scope at any time according to user operations.

[0064] Normally, in order to satisfy the user's gaming experience after turning on the sniper scope, the terminal device can perform AI super-resolution processing on the entire game scene after detecting that the user has turned on the sniper scope, so that the user can see clear images both inside and outside the sniper scope, providing the user with better gaming visual effects.

[0065] However, the above-mentioned image processing method for the entire game scene has the problem of high power consumption. The high power consumption during the game process causes the terminal device to heat up and freeze during the game, affecting the image quality of the game screen and the user's gaming experience.

[0066] In view of this, an embodiment of the present application provides an image processing method that enables a terminal device to perform AI super-resolution processing on images in a first application and within a sniper scope, while the area outside the sniper scope is not subjected to AI super-resolution processing. This improves the clarity of the image within the sniper scope while reducing the power consumption of the device.

[0067] It can be understood that the image processing method provided in the embodiment of the present application can be applied not only to shooting game applications, but also to other game applications containing shooting scenes, or to other applications for increasing the resolution of local areas. The embodiment of the present application does not make specific limitations on this.

[0068] In some embodiments, Figure 2 In the interface shown in a, when the terminal device receives a trigger operation from the user on the small map in the upper right corner, the terminal device can enlarge the small map and improve the resolution of the small map based on the image processing method provided in the embodiment of the present application, so that the user can see the content in the small map more clearly.

[0069] In some embodiments, in a scenario where a user is viewing any image, when the terminal device receives an operation by the user to crop or enlarge an image, the terminal device can improve the resolution of the image based on the image processing method provided in an embodiment of the present application, so that the user can see a clearer image.

[0070] In some embodiments, in a scenario where a user is watching a video, when the terminal device receives a request from the user to zoom in on a local area in the video, the terminal device can improve the resolution of the local area in the video based on the image processing method provided in the embodiment of the present application, so that the user can see a clearer video. In this scenario, the terminal device needs to perform the image processing method provided in the embodiment of the present application on the local area in each frame of the image.

[0071] The above-mentioned terminal devices may be referred to as terminals, user equipment (UE), mobile stations (MS), mobile terminals (MT), etc. For example, the terminal devices may be mobile phones, smart TVs, wearable devices, tablet computers (Pad), computers with wireless transceiver functions, virtual reality (VR) terminal devices, augmented reality (AR) terminal devices, wireless terminals in industrial control, wireless terminals in self-driving, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, etc. The embodiments of the present application do not limit the specific technologies and specific device forms used by the terminal devices.

[0072] In order to better understand the embodiment of the present application, the structure of the terminal device of the embodiment of the present application is introduced below. Figure 3 A schematic diagram of the hardware structure of a terminal device provided in an embodiment of the present application.

[0073] like Figure 3 As shown, the terminal device may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, an indicator 192, a camera 193, and a display screen 194, etc.

[0074] The sensor module 180 may include one or more of the following, for example: a pressure sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, or a bone conduction sensor.

[0075] It is to be understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the terminal device. In other embodiments of the present application, the terminal device may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0076] The processor 110 may include one or more processing units. Different processing units may be independent devices or integrated into one or more processors. The processor 110 may also be provided with a memory for storing instructions and data.

[0077] The USB interface 130 is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the terminal device, and can also be used to transfer data between the terminal device and peripheral devices. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other devices, such as AR devices, etc.

[0078] The charging management module 140 is used to receive charging input from a charger, which may be a wireless charger or a wired charger. The power management module 141 is used to connect the charging management module 140 to the processor 110 .

[0079] The wireless communication function of the terminal device can be implemented through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160 and baseband processor.

[0080] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. The antenna in the terminal device can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of the antenna.

[0081] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to terminal devices. The mobile communication module 150 can include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, etc. the received electromagnetic waves, and transmit them to the modem for demodulation.

[0082] The wireless communication module 160 can provide wireless communication solutions applied to terminal devices, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), etc.

[0083] The modem may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker, a receiver, etc.), or displays an image or video through a display screen 194. In some embodiments, the modem may be an independent device. In other embodiments, the modem may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0084] The terminal device realizes the display function through the GPU, the display screen 194, and the application processor. The GPU is a microprocessor for image processing, connecting the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering.

[0085] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. In some embodiments, the terminal device may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0086] The terminal device can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0087] The camera 193 is used to capture static images or videos. In some embodiments, the terminal device may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0088] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the terminal device. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function. For example, files such as music and videos can be stored in the external memory card.

[0089] The internal memory 121 can be used to store computer executable program codes, and the executable program codes include instructions. The internal memory 121 can include a program storage area and a data storage area.

[0090] The terminal device can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the earphone interface 170D, and the application processor.

[0091] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. Speaker 170A, also known as "speaker", is used to convert audio electrical signals into sound signals. The terminal device can listen to music or listen to hands-free calls through speaker 170A. Receiver 170B, also known as "earpiece", is used to convert audio electrical signals into sound signals. When the terminal device answers a call or voice message, the voice can be answered by placing receiver 170B close to the human ear. Headphone interface 170D is used to connect wired headphones. Microphone 170C, also known as "microphone" or "microphone", is used to convert sound signals into electrical signals. In an embodiment of the present application, the terminal device may have a microphone 170C.

[0092] The button 190 includes a power button, a volume button, etc. The button 190 may be a mechanical button. It may also be a touch button. The terminal device may receive the button input and generate a key signal input related to the user settings and function control of the terminal device. The indicator 192 may be an indicator light, which may be used to indicate the charging status, power change, message, missed call, notification, etc.

[0093] In addition, on top of the above components, the device also runs an operating system, such as an iOS operating system, an Android operating system, or a Windows operating system, etc. Applications can be installed and run on the operating system.

[0094] The software system of the terminal device can adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservice architecture, or a cloud architecture, etc., which will not be elaborated here.

[0095] For example, Figure 4 A schematic diagram of the software structure of a terminal device provided in an embodiment of the present application.

[0096] like Figure 4As shown, the layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into multiple layers, from top to bottom, including the application (APP) layer, the application framework (framework) layer, the native layer, the hardware abstraction layer (HAL), the kernel layer (kernel), and the hardware (hardware) layer, etc., which are not limited in the embodiments of the present application.

[0097] The application layer may include a series of application packages. The application packages may include: shooting game applications, in which the terminal device may execute the image processing method provided in the embodiment of the present application based on the user's operation of opening the sniper scope, so as to improve the resolution of the image in the sniper scope, so as to enable the user to aim at the target in the sniper scope.

[0098] For example, when the shooting game application detects any operation of the user on the shooting game application, the shooting game application can generate scene information corresponding to the any operation, and send the scene information to the game optimization module.

[0099] For example, when a user starts a shooting game application, the shooting game application can send the corresponding identification (such as identification 1) in the game scene corresponding to starting the shooting game application to the game optimization module, so that the game optimization module can clearly understand the current game scene. Or, when a user parachutes in a shooting game application, the shooting game application can send the corresponding identification (such as identification 102) in the game scene corresponding to parachuting to the game optimization module. Or, when a user turns on a sniper scope in a shooting game application, the shooting game application can send the corresponding identification in the game scene corresponding to turning on the sniper scope to the game optimization module.

[0100] Among them, opening different sniper scopes may correspond to different identifiers. For example, when the user opens a 1x scope (or understood as a side scope), the identifier sent by the shooting game application may be 201; when the user opens a 2x scope, the identifier sent by the shooting game application may be 301; when the user opens a 3x scope, the identifier sent by the shooting game application may be 701; when the user opens a 4x scope, the identifier sent by the shooting game application may be 401; when the user opens a 6x scope, the identifier sent by the shooting game application may be 901; when the user opens an 8x scope, the identifier sent by the shooting game application may be 501; when the user opens a tactical scope, the identifier sent by the shooting game application may be 1001.

[0101] It is understandable that the identifier can be a number, a letter, or a combination of numbers and letters, etc. In the embodiment of the present application, there is no specific limitation on the identifier issued by the shooting game application when different scene information is detected.

[0102] In some embodiments, the application layer may also include one or more of the following: Bluetooth, calendar, phone, or video, etc. ( Figure 4 (not shown in the figure), which is not limited in the embodiments of the present application.

[0103] The application framework layer provides application programming interface (API) and programming framework for the applications in the application layer. The application framework layer includes some predefined interfaces.

[0104] The application framework layer may include: a game optimization module (game kit), which can implement the sending of game scene information, for example, sending the scene information obtained from a shooting game application to the game business processing module.

[0105] In some embodiments, the application framework layer may include a window manager, a content provider, a resource manager, a view system, a notification manager, etc. ( Figure 4 not shown).

[0106] Among them, the window manager is used to manage window programs. The window manager can obtain the size of the display screen, determine whether there is a status bar, lock the screen, touch the screen, drag the screen, capture the screen, etc. The content provider is used to store and obtain data and make the data accessible to applications. The data may include video, images, audio, calls made and received, browsing history and bookmarks, phone books, etc. The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build applications. The display interface can be composed of one or more views. For example, a display interface including a text message notification icon can include a view for displaying text and a view for displaying pictures. The resource manager provides various resources for applications, such as localized strings, icons, pictures, layout files, video files, etc.

[0107] The local layer can be used to store modules developed by the terminal device, such as: game service processing module (AGP), graphics acceleration engine (GTX), etc. The local layer can also include a display synthesis system (surface flinger), and the display synthesis system can also be set in the application framework layer. The embodiment of the present application does not make specific limitations on this.

[0108] The game service processing module is used to perform scene recognition, graphics frame rate control, and management functions related to game services. In the embodiment of the present application, the game service processing module can realize the recognition of the game scene, for example, by setting the preset identifier corresponding to the preset scene, determining whether the terminal device meets the preset scene, such as when the game service processing module receives the identifier 901 issued by the game optimization module, and determines that the identifier 901 meets the preset identifier, the game service processing module can determine that the preset scene required by the embodiment of the present application is recognized. Among them, the preset scene can be a scene with a multi-fold mirror used in the game.

[0109] The graphics acceleration engine is used to perform AI super-resolution processing on the image. In the embodiment of the present application, the graphics acceleration engine can obtain a scene image that meets the preset scene, obtain the image inside the zoom lens and the image outside the zoom lens in the scene image by image feature recognition and image segmentation, obtain the image inside the zoom lens after AI super-resolution processing by performing AI super-resolution processing on the image inside the zoom lens, and further fuse the image outside the zoom lens and the image inside the zoom lens after AI super-resolution processing to obtain the final target image.

[0110] The display synthesis system is used to provide layer processing services, responsible for rendering, synthesis and classification of layer contents sent by different modules. In the embodiment of the present application, the display synthesis system can synthesize the target image and the contents of other layers of the terminal device to synthesize the image synthesized by the synthesis layer.

[0111] The purpose of the hardware abstraction layer is to abstract the hardware. It can provide a unified interface for querying hardware devices for upper-layer applications, or it can also provide data storage services for upper-layer applications.

[0112] The hardware abstraction layer may include one or more of the following: a hardware composer (HWC), a GPU user space driver (GPU user space driver), or QNN, etc. Among them, multiple modules in the HAL layer may comply with the hardware abstraction layer interface description language (HAL interface definition language, HIDL).

[0113] The hardware synthesis processor is used to transmit the layer data prepared by the display synthesis system to the required place. In the embodiment of the present application, the hardware synthesis processor can be used to send the layer-synthesized image sent by the display synthesis system to the LCD display driver of the kernel layer.

[0114] The kernel layer is a layer between hardware and software. The kernel layer is used to drive the hardware to make the hardware work. The kernel layer may include one or more of the following, such as a liquid crystal display driver (LCD driver), a GPU driver, an NPU driver, etc., which is not limited in the embodiments of the present application.

[0115] Among them, the LCD display driver is used to provide backlight for the display.

[0116] The hardware layer may include one or more of the following: display, GPU, and NPU, etc.

[0117] It can be understood that the software architecture provided in the embodiment of the present application is only an example, and the terminal device may also include other layers and / or modules, which are not limited in the embodiment of the present application.

[0118] The following specific embodiments are used to describe in detail the technical solution of the present application and how the technical solution of the present application solves the above technical problems. The following specific embodiments can be implemented independently or in combination with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0119] For example, Figure 5 A flowchart of an image processing method provided in an embodiment of the present application.

[0120] like Figure 5 As shown, the image processing method may include the following steps:

[0121] S501: In response to a user starting a shooting game application, a terminal device loads a preset model.

[0122] Among them, the operation of starting a shooting game application can be: the user triggers the function button corresponding to the shooting game application, or it can be the user triggers the thumbnail image corresponding to the shooting game application from the background multi-tasking interface, etc. The specific operation of starting the shooting game application is not limited in the embodiment of the present application.

[0123] The preset model may include a convolutional neural network model with AI super-resolution capability. The preset model is used to perform AI super-resolution processing on low-resolution images, improve the resolution of the image, and output a high-resolution image. The preset model can be set in the device memory and can be called at any time by the terminal device.

[0124] For example, in response to the user's operation of starting a shooting game application, the terminal device may also generate scene information corresponding to starting the shooting game application, and the scene information may include: an identifier 1 corresponding to starting the shooting game application. Further, the terminal device may use the scene information to perform the steps shown in S502.

[0125] S502: The terminal device uses the scene information to perform scene recognition.

[0126] It is understandable that, in response to any operation of the user in the shooting game application, the terminal device can generate scene information corresponding to the operation and use the scene information to perform scene recognition.

[0127] The scene information includes: status information indicating the shooting game application, such as application login and application loading information, and status information generated when the shooting game application runs the game, such as the birth island, parachuting, landing, final circle, sniper scope opening status, etc., and the status information in the scene information can be saved in the terminal device in the form of an identifier.

[0128] For example, after the user starts a shooting game application, the terminal device may generate scene information according to different states of the game. For example, when it is detected that the user turns on the 2x scope, the scene information may include an identifier 201, which is used to instruct the user to turn on the 2x scope.

[0129] Further, the terminal device may perform scene recognition according to the identifier in the scene information, for example, a preset identifier corresponding to a preset scene may be preset in the terminal device, and when it is determined that the currently detected identifier meets the preset identifier, it is determined that the shooting game application meets the preset scene. Among them, the preset scene may be a scene of turning on a multi-fold scope, and the preset identifier corresponding to the preset scene may include one or more of the following, for example: identifier 301, identifier 401, identifier 701, identifier 901, identifier 501 or identifier 1001. Among them, identifier 201 may instruct the user to turn on a 2x scope, identifier 401 is used to instruct the user to turn on a 4x scope, identifier 901 may instruct the user to turn on a 6x scope, identifier 501 is used to instruct the user to turn on an 8x scope, and identifier 1001 is used to instruct the user to turn on a tactical scope.

[0130] For example, Figure 6 A schematic diagram of an interface for opening a sniper scope provided in an embodiment of the present application.

[0131] Figure 6 The sniper scope in the interface shown in a in the figure can be a side view scope. Figure 6 The sniper scope in the interface shown in b can be a red dot scope. Figure 6The sniper scope in the interface shown in c in the figure can be a holographic scope. Since the side view does not need to magnify the content in the lens, and the red dot scope and the holographic scope have a low magnification of the content in the lens, the embodiment of the present application may not be used. Figure 6 The image processing method described in the embodiments of the present application is executed in the three scenarios.

[0132] In a possible implementation, in order to ensure that the user can see a clearer picture in different sniper scopes, the embodiment of the present application can also Figure 6 Perform image processing on the image in the sniper scope to improve the clarity of the image in the sniper scope.

[0133] For example, Figure 7 A schematic diagram of another interface for opening a sniper scope provided in an embodiment of the present application.

[0134] Figure 7 The sniper scope in the interface shown in a can be a tactical scope. Figure 7 The sniper scope in the interface shown in b can be a 2x scope. Figure 7 The sniper scope in the interface shown in c can be a triple scope. Figure 7 The sniper scope in the interface shown in d can be a quadruple scope. Figure 7 The sniper scope in the interface shown in e can be a six-fold scope. Figure 7 The sniper scope in the interface shown in f in the figure can be an eight-fold scope. Among them, the tactical scope can realize the switching of any scope from a four-fold scope to an eight-fold scope, so in Figure 7 In the interface shown in a, a tactical sight with a quadruple scope is used as an example for illustration, and this example does not constitute a limitation on the embodiments of the present application.

[0135] It is understandable that the magnification of the target in different magnification lenses is different. Figure 7 , so Figure 7 The scene where the sniper scope is located is a scene to which the image processing method described in the embodiments of the present application is applied.

[0136] S503: When a preset scene is identified, the terminal device obtains a scene image.

[0137] The scene image may be an image that satisfies a preset scene. For example, the scene image may be a scene image that satisfies a preset scene. Figure 7 The scene image can be obtained by any interface in the scene image. Among them, the image in the zoom lens can be an image obtained by magnifying it using the magnification of the sniper scope.

[0138] In some embodiments, when a preset scene is not recognized, the terminal device can acquire a scene image and instruct the display screen to display the scene image.

[0139] S504: The terminal device performs image feature recognition and image segmentation processing on the scene image to obtain an image inside the zoom lens and an image outside the zoom lens.

[0140] Image feature recognition is used to identify the image within the zoom lens in the scene image. Figure 6 In any interface, since the main feature inside the telescope is a large circle, the terminal device can identify the largest circle in the scene image through graphic feature recognition.

[0141] For example, the terminal device can implement circle detection through the HoughCircles function in opengcv. Specifically, the terminal device can read the scene image through the imread function, and perform image preprocessing, image cropping, adaptiveThreshold filtering and binarization on the scene image in sequence, and then use the HoughCircles function to perform circle recognition to obtain the circle in the scene image.

[0142] Image segmentation is used to segment the scene image into an image within the zoom lens and an image outside the zoom lens. The image segmentation method can be: color image segmentation, grabCut algorithm segmentation, floodFill flood filling segmentation, or watershed segmentation. The specific method of image segmentation is not limited in the embodiments of the present application.

[0143] S505. The terminal device performs AI super-resolution processing on the image in the zoom lens to obtain the image in the zoom lens after AI super-resolution processing.

[0144] The terminal device can input the image in the zoom lens into a preset model with AI super-resolution capability, and output the image in the zoom lens processed by AI super-resolution.

[0145] The number of pixels of the image in the zoom can be a first value, and the number of pixels of the image in the zoom after AI super-resolution processing can be a second value, and the second value is greater than the first value. The preset model may include a 7-layer convolutional neural network. And the pixel density corresponding to the image in the zoom is less than the pixel density corresponding to the image in the zoom after AI super-resolution processing.

[0146] It can be understood that the preset model can default to increasing the number of pixels of a preset multiple for the image in the zoom. For example, when the number of pixels in the image in the zoom is N, the number of pixels in the image in the zoom after AI super-resolution processing can be 1.5N, 2N, 3N or other values.

[0147] In some embodiments, the terminal device may also match a more appropriate number of pixels according to the magnification of the lens, for example, the higher the magnification of the lens, the more pixels are added. For example, the correspondence between the magnification of the lens and the pixel increase multiple can be set in the terminal device, and the pixel increase multiple can be the ratio of the second value to the first value, such as the value obtained by dividing the second value by the first value. For example, when the lens is a 2x lens, the pixel increase multiple can be a value of 2, and when the lens is a 4x lens, the pixel increase multiple can be a value of 3, and when the lens is an 8x lens, the pixel increase multiple can be a value of 5. The embodiment of the present application does not limit the value of the pixel increase multiple. In this way, the terminal device can significantly improve the resolution of the picture under different magnifications and provide users with clearer pictures.

[0148] In the solution of increasing the number of pixels through a preset mode, the terminal device can input the pixel increase multiple and the image in the zoom lens into the preset model, so that the preset model can process the image in the zoom lens according to the pixel increase multiple, and obtain the image in the zoom lens after AI super-resolution processing.

[0149] For example, Figure 8 A schematic diagram of AI super-resolution processing provided in an embodiment of the present application. Figure 8 In the corresponding embodiment, the terminal device can input a low-resolution image into a preset model and output a high-super-resolution image. The low-resolution image can be an image in a zoom lens, and the super-resolution image can be an image in a zoom lens that has been processed by AI super-resolution.

[0150] like Figure 8 As shown, the size of the low-resolution image can be 160*160*3; the terminal device performs differential amplification on the low-resolution image to obtain a differential amplified image, and the size of the differential amplified image can be 240*240*3. The terminal device rearranges the differential amplified image to obtain multiple images of 80*80*27. The terminal device inputs multiple images of 80*80*27 into a 7-layer convolutional neural network model to obtain an output result, and rearranges the output result to obtain multiple images of 80*80*27. The terminal device integrates multiple images of 80*80*27 into a super-resolution image of 240*240*3, which can be the image in the zoom lens after AI super-resolution processing described in the embodiment of the present application. Among them, the model input size of the low-resolution image can be (M, M, 3), M is a positive number greater than 0, and it can be understood that the embodiment of the present application does not limit the image size of the preset model input.

[0151] S506: The terminal device performs image stitching on the image outside the zoom lens and the image inside the zoom lens after AI super-resolution processing to obtain a target image.

[0152] The image stitching method may include one or more of the following steps: image matching, reprojection, stitching, or fusion.

[0153] S507: The terminal device displays the target image.

[0154] For example, the terminal device can display Figure 7 In any interface, Figure 7 The image within the zoom lens of any interface can be a high-resolution image, and the resolution of the image outside the zoom lens does not change.

[0155] Based on this, an embodiment of the present application provides an image processing method, so that the terminal device can perform AI super-resolution processing on the area within the scope through scene recognition, which can not only ensure the user's sniping experience in the game, but also reduce the power consumption caused by AI super-resolution processing.

[0156] exist Figure 5 Based on the corresponding embodiment, in a possible implementation manner, the interactive data interaction process between the modules in the terminal device of the image processing method can be referred to Fig. 9 The corresponding embodiments.

[0157] For example, Fig. 9 A schematic diagram of module interaction of an image processing method provided in an embodiment of the present application. Fig. 9 In the corresponding embodiment, the terminal device may include: a shooting game application, a game optimization module, a game service processing module, a graphics acceleration engine, a display synthesis system, a hardware synthesis processor, a liquid crystal display driver, and a display module. The functions of each module can be found in Figure 4 The description in the corresponding embodiments will not be repeated here.

[0158] like Fig. 9 As shown, the image processing method may include the following steps:

[0159] S901: In response to a user starting a shooting game application, a graphics acceleration engine loads a preset model.

[0160] Exemplarily, the graphics acceleration engine can detect application programs such as shooting game applications, and load a preset model when it is detected that the shooting game application is started.

[0161] S902: In response to the user opening the sniper scope, the game optimization module obtains scene information from the shooting game application.

[0162] TGPA can be integrated into shooting game applications. TGPA is used to connect terminal devices and shooting game applications, and build a bridge between shooting game applications and device manufacturers. For example, terminal devices can obtain various information in shooting game applications, such as scene information, through TGPA.

[0163] Exemplarily, in response to any operation of the user in the shooting game application, the shooting game application can send the scene information corresponding to the operation through TGPA. For example, in response to the user's operation of opening a sniper scope in the shooting game application, the shooting game application can send the scene information corresponding to opening the sniper scope through TGPA, so that the game optimization module can obtain the scene information from the shooting game application.

[0164] It is understandable that, when a user uses a shooting game application, various operations of the user can cause the shooting game application to continuously generate scene information, and send the scene information to the game service processing module through the steps shown in S902-S903.

[0165] S903: The game business processing module obtains scene information from the game optimization module.

[0166] S904: The game service processing module uses the scene information to perform scene recognition.

[0167] The specific process of the game business processing module using the scene information to perform scene recognition can be found in the step shown in S502, which will not be repeated here.

[0168] S905: The graphics acceleration engine obtains a message indicating that a preset scene has been recognized from the game service processing module.

[0169] S906: The graphics acceleration engine obtains a scene image.

[0170] Exemplarily, when the graphics acceleration engine determines that the current game scene satisfies a preset scene, the graphics acceleration engine may obtain a scene image that satisfies the preset scene.

[0171] S907: The graphics acceleration engine performs image feature recognition and image segmentation processing on the scene image to obtain an image inside the zoom lens and an image outside the zoom lens.

[0172] The specific process of image feature recognition and image segmentation processing can refer to the step shown in S504, which will not be repeated here.

[0173] S908. The graphics acceleration engine uses a preset model to perform AI super-resolution processing on the image in the zoom lens to obtain an image in the zoom lens that has been processed by AI super-resolution.

[0174] Exemplarily, the graphics acceleration engine can call QNN and NPU driver, input the image in the zoom lens into the preset model in NPU, and obtain the image in the zoom lens processed by AI super resolution; adaptively, NPU calls NPU driver and QNN, and returns the generated image in the zoom lens processed by AI super resolution to the graphics acceleration engine. The process of generating the image in the zoom lens processed by AI super resolution using the preset model can refer to the steps shown in S505, which will not be repeated here.

[0175] S909, the graphics acceleration engine performs image stitching on the image outside the zoom lens and the image inside the zoom lens after AI super-resolution processing to obtain a target image.

[0176] The method of image stitching can refer to the step shown in S506, which will not be described in detail here.

[0177] In some embodiments, the graphics acceleration engine can also add different pixels to images under different magnifications according to the correspondence between the magnification of the magnification lens and the pixel increase multiple, so as to improve the clarity under the high magnification lens. Among them, the correspondence between the magnification of the magnification lens and the pixel increase multiple can be set in the game business processing module or the graphics acceleration engine and other modules, which is not limited in the embodiments of the present application.

[0178] S910: The display composition system obtains a target image from a graphics acceleration engine.

[0179] S911. The display synthesis system performs layer synthesis on the target image and other layers to obtain a layer-synthesized image.

[0180] Among them, the other layers may include one or more of the following: layers corresponding to other applications except shooting game applications, layers corresponding to the top status bar of the terminal device, or layers corresponding to applications in the sidebar, etc., which are not limited in the embodiments of the present application.

[0181] S912: The hardware synthesis processor obtains the layer-synthesized image from the display synthesis system.

[0182] S913. The LCD display driver obtains the layer-synthesized image from the hardware synthesis processor.

[0183] S914: The display obtains the layer-synthesized image from the liquid crystal display driver.

[0184] S915. The display shows the image after the layers are synthesized.

[0185] For example, the display can show Figure 7 Any interface in .

[0186] It is understandable that the image processing method provided in the embodiment of the present application can not only move the load on the GPU to the NPU for execution based on the self-developed AI super-resolution algorithm for mobile game effects, but also make full use of the underlying capabilities provided by the chip manufacturer to greatly improve the user's image quality experience or reduce the heat generation during the user's game. It can also achieve local super-resolution by introducing scene recognition, image feature recognition, image segmentation and image fusion technology, which can not only improve the resolution of local scenes that users are concerned about, but also reduce the power consumption problem caused by super-resolution of non-critical scenes.

[0187] It should be understood that the interface provided in the embodiment of the present application is merely an example and does not constitute a limitation on the embodiment of the present application.

[0188] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0189] Combined with the above Figure 5-Figure 9 , the method provided in the embodiment of the present application is described, and the device for executing the above method provided in the embodiment of the present application is described below. Fig.10 As shown, Fig.10 A structural schematic diagram of an image processing device provided in an embodiment of the present application. The image processing device may be a terminal device in an embodiment of the present application, or a chip or chip system in a terminal device.

[0190] like Fig.10 As shown, the image processing apparatus 1000 can be used in a communication device, a circuit, a hardware component or a chip, and the image processing apparatus 1000 includes: a display unit 1001, a processing unit 1002, and a communication unit 1003, etc. Among them, the display unit 1001 is used to support the display step performed by the image processing method; the processing unit 1002 is used to support the image processing apparatus 1000 to perform the information processing step; and the communication unit 1003 is used to support the image processing apparatus 1000 to perform the steps of receiving or sending messages.

[0191] Specifically, the processing unit 1002 and the display unit 1001 may be integrated together, and the processing unit 1002 and the display unit 1001 may communicate with each other.

[0192] In a possible implementation, the image processing apparatus 1000 may further include: a storage unit 1004. The storage unit 1004 may include one or more memories, and the memory may be a device in one or more devices or circuits for storing programs or data.

[0193] The storage unit 1004 may exist independently and be connected to the processing unit 1002 via a communication bus. The storage unit 1004 may also be integrated with the processing unit 1002.

[0194] Taking the image processing device 1000 as an example, which can be a chip or chip system of a terminal device in an embodiment of the present application, the storage unit 1004 can store computer execution instructions of the method of the terminal device, so that the processing unit 1002 executes the method of the terminal device in the above embodiment. The storage unit 1004 can be a register, a cache, or a random access memory (RAM), etc. The storage unit 1004 can be integrated with the processing unit 1002. The storage unit 1004 can be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, and the storage unit 1004 can be independent of the processing unit 1002.

[0195] In a possible implementation, the image processing apparatus 1000 may further include: a communication unit 1003. The communication unit 1003 is used to support the image processing apparatus 1000 to interact with other devices. Exemplarily, when the image processing apparatus 1000 is a terminal device, the communication unit 1003 may be a communication interface or an interface circuit. When the image processing apparatus 1000 is a chip or a chip system in a terminal device, the communication unit 1003 may be a communication interface. For example, the communication interface may be an input / output interface, a pin or a circuit, etc.

[0196] The device of this embodiment can be used to execute the steps executed in the above method embodiment. Its implementation principle and technical effects are similar and will not be described in detail here.

[0197] For example, Fig.11 A schematic diagram of the hardware structure of another terminal device provided in an embodiment of the present application.

[0198] The terminal device includes a processor 1101, a communication line 1104 and at least one communication interface ( Fig.11 The communication interface 1103 is used as an example for explanation).

[0199] The processor 1101 may be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the present application.

[0200] Communications link 1104 may include circuitry to transmit information between the above-described components.

[0201] The communication interface 1103 uses any transceiver or other device for communicating with other devices or communication networks, such as Ethernet, wireless local area networks (WLAN), etc.

[0202] Possibly, the terminal device may further include a memory 1102 .

[0203] The memory 1102 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory may exist independently and be connected to the processor via a communication line 1104. The memory may also be integrated with the processor.

[0204] The memory 1102 is used to store computer-executable instructions for executing the solution of the present application, and the execution is controlled by the processor 1101. The processor 1101 is used to execute the computer-executable instructions stored in the memory 1102, thereby implementing the method provided by the embodiment of the present application.

[0205] Possibly, the computer-executable instructions in the embodiments of the present application may also be referred to as application code, and the embodiments of the present application do not specifically limit this.

[0206] In a specific implementation, as an embodiment, the processor 1101 may include one or more CPUs, such as Fig.11 CPU0 and CPU1 in.

[0207] In a specific implementation, as an embodiment, the terminal device may include multiple processors, such as Fig.11 1 and 1105. Each of these processors may be a single-CPU processor or a multi-CPU processor. A processor herein may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0208] In the above embodiments, the instructions stored in the memory for execution by the processor may be implemented in the form of a computer program product, wherein the computer program product may be pre-written in the memory, or may be downloaded and installed in the memory in the form of software.

[0209] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a server or data center that includes one or more available media integrated. For example, the available medium can include a magnetic medium (e.g., a floppy disk, a hard disk or a tape), an optical medium (e.g., a digital versatile disc (digital versatiledisc, DVD)), or a semiconductor medium (e.g., a solid state drive (solid state disk, SSD)), etc.

[0210] The embodiments of the present application also provide a computer-readable storage medium. The methods described in the above embodiments can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. Computer-readable media may include computer storage media and communication media, and may also include any medium that can transfer a computer program from one place to another. The storage medium may be any target medium that can be accessed by a computer.

[0211] As a possible design, the computer readable medium may include a compact disc read-only memory (CD-ROM), RAM, ROM, EEPROM or other optical disc storage; the computer readable medium may include a magnetic disk storage or other magnetic disk storage device. Moreover, any connection line may also be appropriately referred to as a computer readable medium. For example, if the software is transmitted from a website, server or other remote source using a coaxial cable, fiber optic cable, twisted pair, DSL or wireless technology (such as infrared, radio and microwave), the coaxial cable, fiber optic cable, twisted pair, DSL or wireless technology such as infrared, radio and microwave are included in the definition of medium. Disk and disc as used herein include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, where disks usually reproduce data magnetically, while optical discs reproduce data optically using lasers.

[0212] The above combinations should also be included in the scope of computer-readable media. The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. An image processing method, characterized in that: The method comprises: The terminal device receives a first operation for the first application, In response to the first operation, the terminal device displays a first interface, wherein the first interface includes: a first button for turning on a sniper scope; The terminal device receives a second operation on the first button; In response to the second operation, the terminal device obtains a first image in the first application, where the first image includes an image within the sniper scope and an image outside the sniper scope, and the number of pixels of the image within the sniper scope is a first value; The terminal device inputs the image in the sniper scope into a preset model and outputs a second image, wherein the preset model is used to perform AI super-resolution processing on the image in the sniper scope, and the number of pixels in the second image is a second value, and the second value is greater than the first value; The terminal device stitches the second image and the image outside the sniper scope into a third image; The terminal device displays the third image.

2. The method according to claim 1, characterized in that a correspondence between the magnification of the sniper scope preset in the terminal device and the pixel increase multiple, wherein the pixel increase multiple is the ratio of the second value to the first value; The terminal device inputs the image in the sniper scope into a preset model and outputs a second image, including: When the magnification of the sniper scope is a third value, the terminal device obtains a fourth value corresponding to the third value from the corresponding relationship, wherein the corresponding relationship includes a corresponding relationship between the third value and the fourth value; The terminal device inputs the fourth value and the image in the sniper scope into the preset model, and outputs the second image, wherein the ratio of the second value to the first value is the fourth value.

3. The method according to claim 1 or 2, characterized in that: In response to the second operation, the terminal device acquires a first image in the first application, including: In response to the second operation, the terminal device generates a first identifier corresponding to the first operation, where the first identifier is an identifier corresponding to opening a sniper scope; When the terminal device determines that the first identifier satisfies a preset identifier, the terminal device obtains the first image in the first application.

4. The method according to any one of claims 1 to 3, characterized in that: After the terminal device acquires the first image in the first application, the method further includes: The terminal device recognizes at least one circle in the first image through image feature recognition; The terminal device determines the largest circle among the at least one circle, and the content displayed in the largest circle is the image in the sniper scope; The terminal device divides the first image into an image within the sniper scope and an image outside the sniper scope.

5. The method according to any one of claims 1 to 4, characterized in that: The terminal device includes an NPU, The terminal device inputs the image in the sniper scope into a preset model and outputs a second image, including: the terminal device uses the NPU to input the image in the sniper scope into the preset model and output the second image.

6. The method according to any one of claims 1 to 5, characterized in that: Before the terminal device receives the first operation for the first application, the method further includes: The terminal device receives a third operation for the first application, where the third operation is used to start the first application; In response to the third operation, the terminal device loads the preset model.

7. The method according to claim 2, characterized in that The first application integrates TGPA, and the terminal device includes: a game optimization module and a game service processing module. The terminal device generates a first identifier corresponding to the first operation, including: the first application generates a first identifier corresponding to the first operation through the TGPA, and sends the first identifier to the game optimization module; the game optimization module sends the first identifier to the game service processing module; The terminal device determines that the first identifier satisfies a preset identifier, including: the game service processing module determines that the first identifier satisfies the preset identifier.

8. The method according to claim 7, characterized in that The terminal device also includes: a graphics acceleration engine, The method further includes: the graphics acceleration engine dividing the first image into an image within the sniper scope and an image outside the sniper scope; The terminal device stitches the second image and the image outside the sniper scope into a third image, including: the graphics acceleration engine stitches the second image and the image outside the sniper scope into the third image.

9. The method according to claim 7 or 8, characterized in that: The terminal device also includes: a display synthesis system, a hardware synthesis processor, a liquid crystal display driver, and a display, and the method also includes: The display synthesis system synthesizes the third image with at least one layer to obtain a fourth image; The display synthesis system sends the fourth image to the display via the hardware synthesis processor and the liquid crystal display driver; The display displays the fourth image.

10. The method according to any one of claims 1 to 9, characterized in that: The first application is a shooting game application, and the sniper scope includes one or more of the following: a 2x scope, a 3x scope, a 4x scope, a 6x scope, an 8x scope, or a tactical sight.

11. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the terminal device executes the method according to any one of claims 1 to 10.

12. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the computer is caused to perform the method according to any one of claims 1 to 10.

Images